Jul 28 15:33:31.438464 ip-10-0-137-14 systemd[1]: Starting Kubernetes Kubelet... Jul 28 15:33:31.859774 ip-10-0-137-14 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:31.859774 ip-10-0-137-14 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 15:33:31.859774 ip-10-0-137-14 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:31.859774 ip-10-0-137-14 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 15:33:31.859774 ip-10-0-137-14 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 15:33:31.860787 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.860690 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 15:33:31.866687 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866669 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.866687 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866686 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.866687 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866690 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866693 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866697 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866701 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866703 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866706 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866709 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866712 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866715 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866718 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866720 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866723 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866725 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866730 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866732 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866735 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866737 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866740 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866742 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866745 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.866779 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866748 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866750 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866753 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866755 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866758 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866760 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866763 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866765 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866768 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866771 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866773 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866776 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866778 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866781 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866784 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866787 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866791 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866793 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866796 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866798 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.867263 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866801 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866803 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866806 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866808 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866811 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866813 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866816 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866820 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866824 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866827 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866830 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866833 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866835 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866838 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866841 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866844 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866847 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866850 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866852 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.867736 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866855 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866858 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866860 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866863 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866865 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866868 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866872 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866876 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866878 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866881 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866885 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866888 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866890 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866893 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866897 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866900 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866903 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866906 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866909 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866911 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.868190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866914 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866918 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866920 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866923 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866926 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866928 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866931 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866934 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866937 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866940 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866943 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866946 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866949 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866951 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866953 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866956 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866959 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.866962 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867697 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.868694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867705 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867708 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867711 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867714 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867717 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867721 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867725 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867727 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867730 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867732 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867735 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867737 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867740 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867743 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867745 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867748 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867751 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867754 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867756 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867759 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.869156 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867762 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867765 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867768 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867770 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867773 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867775 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867778 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867781 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867785 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867789 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867791 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867795 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867797 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867800 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867803 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867805 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867808 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867813 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867816 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.869658 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867818 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867821 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867824 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867826 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867828 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867831 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867833 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867836 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867838 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867840 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867843 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867845 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867848 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867851 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867854 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867856 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867859 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867863 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867866 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.870125 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867869 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867872 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867875 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867877 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867880 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867884 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867886 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867889 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867891 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867894 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867896 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867898 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867902 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867904 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867907 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867909 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867912 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867914 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867917 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867919 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.870604 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867922 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867924 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867927 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867929 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867931 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867935 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867938 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867940 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867943 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867946 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867948 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867951 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867953 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867956 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867959 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867961 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867963 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867966 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867969 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.867971 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.871097 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868455 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868461 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868463 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868466 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868470 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868473 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868476 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868478 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868481 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868483 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868486 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868488 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868491 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868493 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868496 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868498 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868501 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868503 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868506 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868508 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.871598 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868511 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868513 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868516 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868518 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868521 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868523 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868526 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868528 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868531 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868534 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868536 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868539 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868541 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868544 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868546 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868548 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868551 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868554 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868557 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.872086 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868559 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868561 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868564 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868566 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868569 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868572 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868576 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868580 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868583 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868586 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868588 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868591 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868594 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868597 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868599 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868602 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868604 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868606 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868609 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868611 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.872567 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868614 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868616 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868619 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868622 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868624 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868629 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868632 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868635 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868637 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868640 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868642 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868645 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868647 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868650 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868652 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868655 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868657 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868659 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868662 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868664 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.873058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868666 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868669 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868671 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868677 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868679 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868682 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868684 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868687 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868689 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868692 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868694 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868697 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868700 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868702 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868707 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868710 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868713 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868716 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868719 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.868721 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.873588 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868800 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868810 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868820 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868826 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868833 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868837 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868841 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868845 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868848 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868851 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868854 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868857 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868861 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868863 2580 flags.go:64] FLAG: --cgroup-root="" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868866 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868871 2580 flags.go:64] FLAG: --client-ca-file="" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868873 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868877 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868881 2580 flags.go:64] FLAG: --cluster-domain="" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868884 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868887 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868890 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868893 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868897 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 15:33:31.874076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868899 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868902 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868906 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868908 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868911 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868914 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868917 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868920 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868924 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868927 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868929 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868932 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868935 2580 flags.go:64] FLAG: --enable-server="true" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868939 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868944 2580 flags.go:64] FLAG: --event-burst="100" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868946 2580 flags.go:64] FLAG: --event-qps="50" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868949 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868953 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868955 2580 flags.go:64] FLAG: --eviction-hard="" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868959 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868961 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868964 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868968 2580 flags.go:64] FLAG: --eviction-soft="" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868972 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868974 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 15:33:31.874666 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868978 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868980 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868983 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868986 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868988 2580 flags.go:64] FLAG: --feature-gates="" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868993 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868996 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.868998 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869002 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869005 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869007 2580 flags.go:64] FLAG: --help="false" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869010 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-137-14.ec2.internal" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869013 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869016 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869019 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869022 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869025 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869028 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869031 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869033 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869036 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869040 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869043 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869045 2580 flags.go:64] FLAG: --kube-reserved="" Jul 28 15:33:31.875339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869050 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869052 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869055 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869058 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869061 2580 flags.go:64] FLAG: --lock-file="" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869063 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869067 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869071 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869076 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869078 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869081 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869084 2580 flags.go:64] FLAG: --logging-format="text" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869086 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869089 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869092 2580 flags.go:64] FLAG: --manifest-url="" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869095 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869099 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869102 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869106 2580 flags.go:64] FLAG: --max-pods="110" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869109 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869112 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869114 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869117 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869120 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869123 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 15:33:31.875963 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869125 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869133 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869135 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869138 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869141 2580 flags.go:64] FLAG: --pod-cidr="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869145 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869150 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869153 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869156 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869159 2580 flags.go:64] FLAG: --port="10250" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869163 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869166 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-025771abc89cc0c27" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869169 2580 flags.go:64] FLAG: --qos-reserved="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869172 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869175 2580 flags.go:64] FLAG: --register-node="true" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869178 2580 flags.go:64] FLAG: --register-with-taints="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869181 2580 flags.go:64] FLAG: --registry-burst="10" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869184 2580 flags.go:64] FLAG: --registry-qps="5" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869187 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869189 2580 flags.go:64] FLAG: --reserved-memory="" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869193 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869196 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869199 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869215 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869218 2580 flags.go:64] FLAG: --runonce="false" Jul 28 15:33:31.876733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869221 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869224 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869227 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869230 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869233 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869236 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869239 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869242 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869244 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869247 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869250 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869253 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869256 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869259 2580 flags.go:64] FLAG: --system-cgroups="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869262 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869268 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869270 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869273 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869277 2580 flags.go:64] FLAG: --tls-min-version="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869279 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869282 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869286 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869290 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869293 2580 flags.go:64] FLAG: --v="2" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869297 2580 flags.go:64] FLAG: --version="false" Jul 28 15:33:31.877708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869301 2580 flags.go:64] FLAG: --vmodule="" Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869304 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.869308 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869432 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869436 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869440 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869442 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869445 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869447 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869450 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869452 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869455 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869457 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869460 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869462 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869465 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869468 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869470 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869473 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.878812 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869475 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869479 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869482 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869485 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869487 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869489 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869492 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869495 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869497 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869501 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869504 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869506 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869508 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869511 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869513 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869516 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869518 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869521 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869523 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869526 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.879702 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869528 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869531 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869534 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869536 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869539 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869542 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869544 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869548 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869551 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869554 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869557 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869560 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869562 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869566 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869569 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869572 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869575 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869577 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869580 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.880562 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869583 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869585 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869589 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869591 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869594 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869596 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869598 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869601 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869604 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869606 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869608 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869611 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869614 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869616 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869619 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869622 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869626 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869629 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869631 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869634 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.881078 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869636 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869639 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869641 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869643 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869646 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869648 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869652 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869654 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869657 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869659 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869662 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869666 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869670 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869673 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869677 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869680 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869683 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869686 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869688 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869691 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.881804 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869693 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869696 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869698 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.869701 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.870272 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.878908 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.878930 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.878962 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.878976 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879065 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879073 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879080 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879087 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879093 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.882331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879098 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879102 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879107 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879112 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879117 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879122 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879126 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879131 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879135 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879140 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879144 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879148 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879152 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879156 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879161 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879167 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879172 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879177 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879182 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.882733 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879188 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879194 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879218 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879225 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879229 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879234 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879238 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879243 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879247 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879251 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879255 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879260 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879264 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879268 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879272 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879277 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879281 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879285 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879289 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879293 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.883192 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879297 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879301 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879305 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879310 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879314 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879318 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879322 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879326 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879330 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879335 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879339 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879343 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879348 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879352 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879356 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879360 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879364 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879369 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879374 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879378 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879383 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.884030 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879387 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879392 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879396 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879400 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879404 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879408 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879412 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879416 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879420 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879425 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879429 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879433 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879437 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879441 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879455 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879460 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879464 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879468 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879472 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879477 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.884636 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879482 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879486 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879490 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879494 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879498 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879502 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879506 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879510 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879515 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879519 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879524 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879528 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879532 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879536 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.885283 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.879544 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879740 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879750 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879755 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879759 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879764 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879769 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879773 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879778 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879782 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879787 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879791 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879796 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879808 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879814 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879821 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879825 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879830 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879834 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879839 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 15:33:31.885648 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879843 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879849 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879855 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879859 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879864 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879897 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879904 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879910 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879914 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879920 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879924 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879929 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879933 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879937 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879942 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879947 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879951 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879956 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879960 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879964 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 15:33:31.886137 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879968 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879972 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879976 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879980 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879984 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.879988 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880000 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880005 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880009 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880013 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880017 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880021 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880025 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880029 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880033 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880038 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880042 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880046 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880050 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 15:33:31.886683 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880054 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880058 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880062 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880066 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880070 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880074 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880079 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880083 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880087 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880094 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880099 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880104 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880109 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880113 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880118 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880122 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880126 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880131 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880135 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 15:33:31.887168 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880140 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880145 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880150 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880154 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880158 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880163 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880167 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880172 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880176 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880180 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880184 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880188 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880193 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880197 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880219 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880223 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880228 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880232 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880237 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880241 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 15:33:31.887829 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880245 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:31.880249 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.880257 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.880417 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.884834 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.886621 2580 server.go:1019] "Starting client certificate rotation" Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.886728 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 15:33:31.888353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.886769 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 15:33:31.890920 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.890902 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 15:33:31.892102 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.892085 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 15:33:31.902095 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.902075 2580 log.go:25] "Validated CRI v1 runtime API" Jul 28 15:33:31.905653 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.905638 2580 log.go:25] "Validated CRI v1 image API" Jul 28 15:33:31.907242 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.907226 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 15:33:31.909865 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.909846 2580 fs.go:135] Filesystem UUIDs: map[05c093fe-d69b-4a93-a838-0f650888b7a8:/dev/nvme0n1p3 6d94f5dc-83e6-447d-a343-70971b2da7b0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 28 15:33:31.909935 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.909864 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 15:33:31.915247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.915124 2580 manager.go:216] Machine: {Timestamp:2026-07-28 15:33:31.913827768 +0000 UTC m=+0.357479281 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100053 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d3d93d6600645c2d581058ca15428 SystemUUID:ec2d3d93-d660-0645-c2d5-81058ca15428 BootID:240e445b-a9be-4118-94d5-5ca802247aa1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:77:0c:b1:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:77:0c:b1:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:78:3d:fd:fa:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 15:33:31.915247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.915237 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 15:33:31.915409 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.915317 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 15:33:31.918495 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.918467 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 15:33:31.919684 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.918299 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 15:33:31.919684 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.918567 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 15:33:31.921607 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.921580 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 15:33:31.921762 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.921753 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 15:33:31.921812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.921771 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 15:33:31.922407 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.922395 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 15:33:31.923787 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.923777 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 15:33:31.927417 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927404 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 28 15:33:31.927537 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927527 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 15:33:31.927577 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927558 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 28 15:33:31.927577 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927569 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 15:33:31.927646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927582 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 15:33:31.927646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927591 2580 kubelet.go:398] "Adding apiserver pod source" Jul 28 15:33:31.927646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927606 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 15:33:31.927995 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.927983 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 15:33:31.928030 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.928010 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 15:33:31.930291 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.930272 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 15:33:31.931598 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.931584 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 15:33:31.931654 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.931614 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 15:33:31.932314 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932303 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 15:33:31.932351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932318 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 15:33:31.932351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932325 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 15:33:31.932351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932330 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 15:33:31.932351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932340 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 15:33:31.932351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932345 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932369 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932374 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932381 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932386 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932394 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 15:33:31.932487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.932402 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 15:33:31.933154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.933144 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 15:33:31.933154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.933154 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 15:33:31.936453 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.936438 2580 server.go:1268] "Started kubelet" Jul 28 15:33:31.936549 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.936505 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 15:33:31.936590 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.936534 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 15:33:31.936646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.936632 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 15:33:31.937297 ip-10-0-137-14 systemd[1]: Started Kubernetes Kubelet. Jul 28 15:33:31.938531 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.938427 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 15:33:31.938818 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.938802 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 28 15:33:31.940689 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.940672 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 15:33:31.940813 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.940796 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 15:33:31.940873 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.940830 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 15:33:31.943125 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.943107 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 15:33:31.946430 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.946412 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 15:33:31.946500 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.946429 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 15:33:31.946574 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.946552 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:31.946727 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.945634 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c67df949bcabd1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-07-28 15:33:31.936410577 +0000 UTC m=+0.380062090,LastTimestamp:2026-07-28 15:33:31.936410577 +0000 UTC m=+0.380062090,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Jul 28 15:33:31.947396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947243 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 15:33:31.947396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947317 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 15:33:31.947396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947331 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 15:33:31.947396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947374 2580 factory.go:55] Registering systemd factory Jul 28 15:33:31.947396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947392 2580 factory.go:223] Registration of the systemd container factory successfully Jul 28 15:33:31.947599 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947422 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 28 15:33:31.947599 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947447 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 28 15:33:31.948101 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947937 2580 factory.go:153] Registering CRI-O factory Jul 28 15:33:31.948101 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.947953 2580 factory.go:223] Registration of the crio container factory successfully Jul 28 15:33:31.948101 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.948002 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 15:33:31.948101 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.948023 2580 factory.go:103] Registering Raw factory Jul 28 15:33:31.948101 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.948037 2580 manager.go:1196] Started watching for new ooms in manager Jul 28 15:33:31.948726 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.948709 2580 manager.go:349] Starting recovery of all containers Jul 28 15:33:31.958871 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.958840 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 15:33:31.958871 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.958838 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 15:33:31.961890 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.961750 2580 manager.go:354] Recovery completed Jul 28 15:33:31.965763 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.965750 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:31.967951 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.967932 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:31.968018 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.967967 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:31.968018 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.967978 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:31.968487 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.968475 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 15:33:31.968535 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.968487 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 15:33:31.968535 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.968496 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:31.968535 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.968508 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 28 15:33:31.969228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.969188 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:31.969358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.969232 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:31.969358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.969246 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:31.970050 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.970038 2580 policy_none.go:49] "None policy: Start" Jul 28 15:33:31.970090 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.970054 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 15:33:31.970090 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.970065 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 15:33:31.970468 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.970402 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c67df94b9df6fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-07-28 15:33:31.967952634 +0000 UTC m=+0.411604148,LastTimestamp:2026-07-28 15:33:31.967952634 +0000 UTC m=+0.411604148,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Jul 28 15:33:31.971154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.971143 2580 policy_none.go:47] "Start" Jul 28 15:33:31.982028 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.981933 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c67df94b9e450a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-14.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-07-28 15:33:31.967972618 +0000 UTC m=+0.411624132,LastTimestamp:2026-07-28 15:33:31.967972618 +0000 UTC m=+0.411624132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Jul 28 15:33:31.983381 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.983364 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj7vn" Jul 28 15:33:31.990842 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.990767 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c67df94b9e6ab6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-14.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-07-28 15:33:31.967982262 +0000 UTC m=+0.411633775,LastTimestamp:2026-07-28 15:33:31.967982262 +0000 UTC m=+0.411633775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Jul 28 15:33:31.995448 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:31.995429 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj7vn" Jul 28 15:33:31.999786 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:31.999730 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-14.ec2.internal.18c67df94b9df6fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c67df94b9df6fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-07-28 15:33:31.967952634 +0000 UTC m=+0.411604148,LastTimestamp:2026-07-28 15:33:31.969216344 +0000 UTC m=+0.412867865,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Jul 28 15:33:32.004768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.004754 2580 manager.go:337] "Starting Device Plugin manager" Jul 28 15:33:32.004832 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.004785 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 15:33:32.004832 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.004798 2580 server.go:87] "Starting device plugin registration server" Jul 28 15:33:32.005060 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.005048 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 15:33:32.005113 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.005064 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 15:33:32.005302 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.005283 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 15:33:32.005391 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.005377 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 15:33:32.005391 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.005391 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 15:33:32.006473 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.006453 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 15:33:32.006551 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.006515 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.088378 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.088332 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 15:33:32.090992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.089627 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 15:33:32.090992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.089659 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 15:33:32.090992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.089680 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 15:33:32.090992 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.089710 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 15:33:32.092312 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.092292 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:32.105317 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.105303 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:32.106724 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.106709 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:32.106795 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.106741 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:32.106795 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.106754 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:32.106795 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.106777 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.121297 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.121256 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.121297 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.121278 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-14.ec2.internal\": node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.148240 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.148199 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.190187 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.190161 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal"] Jul 28 15:33:32.190288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.190245 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:32.192063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.191349 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:32.192063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.191386 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:32.192063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.191399 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:32.192998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.192984 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:32.193108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.193092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.193165 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.193123 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:32.194960 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.194945 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:32.195064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.194970 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:32.195064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.194948 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:32.195064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.195009 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:32.195064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.195027 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:32.195064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.194980 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:32.196358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.196345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.196423 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.196369 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 15:33:32.197440 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.197425 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Jul 28 15:33:32.197518 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.197450 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 15:33:32.197518 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.197459 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Jul 28 15:33:32.223018 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.222993 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-14.ec2.internal\" not found" node="ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.227288 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.227273 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-14.ec2.internal\" not found" node="ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.248830 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.248807 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.248909 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.248846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.248909 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.248868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.248909 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.248883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/372e08fc4324f245307306950cbd07f2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"372e08fc4324f245307306950cbd07f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349255 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.349224 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.349362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/372e08fc4324f245307306950cbd07f2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"372e08fc4324f245307306950cbd07f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/372e08fc4324f245307306950cbd07f2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"372e08fc4324f245307306950cbd07f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.349464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.349372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfc96076c26aaf99e282a8d26fbce05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"acfc96076c26aaf99e282a8d26fbce05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.449948 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.449859 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.526103 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.526075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.530728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.530674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:32.550551 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.550523 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.650987 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.650937 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.751494 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:32.751406 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:32.766384 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:32.766363 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:34.614288 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:34.614244 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:34.648433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.648407 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 15:33:34.648992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.648584 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 15:33:34.648992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.648583 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 15:33:34.714638 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:34.714605 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Jul 28 15:33:34.762563 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.762494 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:28:33 +0000 UTC" deadline="2028-04-30 10:10:24.468332256 +0000 UTC" Jul 28 15:33:34.762563 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.762561 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15402h36m49.705776401s" Jul 28 15:33:34.797991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.797923 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:34.809862 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.809836 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Jul 28 15:33:34.828124 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:34.828089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc96076c26aaf99e282a8d26fbce05.slice/crio-1313f5e22003507a8b633a14839b75a09fdb5acc39ea938b9623c4ec1be8b30e WatchSource:0}: Error finding container 1313f5e22003507a8b633a14839b75a09fdb5acc39ea938b9623c4ec1be8b30e: Status 404 returned error can't find the container with id 1313f5e22003507a8b633a14839b75a09fdb5acc39ea938b9623c4ec1be8b30e Jul 28 15:33:34.828676 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:34.828657 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372e08fc4324f245307306950cbd07f2.slice/crio-4cae46db88df2a116fbce60a525d40f392c41fa38184e4b168c132b368400632 WatchSource:0}: Error finding container 4cae46db88df2a116fbce60a525d40f392c41fa38184e4b168c132b368400632: Status 404 returned error can't find the container with id 4cae46db88df2a116fbce60a525d40f392c41fa38184e4b168c132b368400632 Jul 28 15:33:34.831210 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.831191 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 15:33:34.832173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.832157 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Jul 28 15:33:34.834049 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.834027 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:33:34.846817 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.846787 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 15:33:34.855590 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.855546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"acfc96076c26aaf99e282a8d26fbce05","Type":"ContainerStarted","Data":"1313f5e22003507a8b633a14839b75a09fdb5acc39ea938b9623c4ec1be8b30e"} Jul 28 15:33:34.856545 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:34.856517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" event={"ID":"372e08fc4324f245307306950cbd07f2","Type":"ContainerStarted","Data":"4cae46db88df2a116fbce60a525d40f392c41fa38184e4b168c132b368400632"} Jul 28 15:33:35.261158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.261075 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:35.691381 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.691288 2580 apiserver.go:52] "Watching apiserver" Jul 28 15:33:35.700616 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.700589 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 15:33:35.701119 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.701097 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vbs4j","openshift-multus/network-metrics-daemon-gs4v8","openshift-network-diagnostics/network-check-target-hxplb","kube-system/global-pull-secret-syncer-cbcgv","kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal","openshift-dns/node-resolver-xz94w","openshift-multus/multus-xpdcx","openshift-network-operator/iptables-alerter-rthgn","openshift-ovn-kubernetes/ovnkube-node-7hv4d","kube-system/konnectivity-agent-fvkrk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs","openshift-cluster-node-tuning-operator/tuned-qmn8h","openshift-image-registry/node-ca-85bv6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal"] Jul 28 15:33:35.702871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.702851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.704327 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.704277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.705625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.705606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.706168 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.706147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 15:33:35.706270 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.706218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 15:33:35.706270 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.706226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-csbcc\"" Jul 28 15:33:35.707374 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.707353 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.708877 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.707555 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 15:33:35.708977 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.708909 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fm7f2\"" Jul 28 15:33:35.709038 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.708999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.709112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709096 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.709223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709197 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.709329 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709225 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 15:33:35.709329 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7tjn\"" Jul 28 15:33:35.709458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709356 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 15:33:35.709458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.709593 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.709577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.712502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.712101 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 15:33:35.712502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.712126 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.712502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.712320 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.712502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.712356 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 15:33:35.713340 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.713323 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 15:33:35.713426 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.713340 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 15:33:35.713426 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.713357 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8qtkq\"" Jul 28 15:33:35.713596 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.713581 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2smfp\"" Jul 28 15:33:35.717716 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.717699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-var-lib-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-netd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-agent-certs\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6vm\" (UniqueName: \"kubernetes.io/projected/c613a05c-0bc3-4bfa-ac92-7ff568727e95-kube-api-access-8d6vm\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-cnibin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-kubelet\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e25eba0e-6332-4616-84e8-24ad2a645237-ovn-node-metrics-cert\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zqx\" (UniqueName: \"kubernetes.io/projected/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-kube-api-access-l8zqx\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c613a05c-0bc3-4bfa-ac92-7ff568727e95-host-slash\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-system-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/5cc89da4-ef78-4830-9c8d-6e53ff954440-kube-api-access-n5fcq\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-slash\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.719998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-host\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-cnibin\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.719999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-systemd-units\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-node-log\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-bin\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-env-overrides\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-script-lib\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c613a05c-0bc3-4bfa-ac92-7ff568727e95-iptables-alerter-script\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-k8s-cni-cncf-io\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-multus-certs\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-ovn\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.720544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-conf-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.720981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-netns\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-systemd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-config\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflp4\" (UniqueName: \"kubernetes.io/projected/e25eba0e-6332-4616-84e8-24ad2a645237-kube-api-access-bflp4\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-serviceca\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-system-cni-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-os-release\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-netns\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-etc-kubernetes\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-etc-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-konnectivity-ca\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.721418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5xh\" (UniqueName: \"kubernetes.io/projected/4ee68990-196b-47a9-bf0a-397cc21718da-kube-api-access-gl5xh\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.721745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-cni-binary-copy\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-socket-dir-parent\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-multus\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-daemon-config\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-log-socket\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.721991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-os-release\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.721991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-bin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.721991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721918 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 15:33:35.721991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-hostroot\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.722131 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.722058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-kubelet\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.722190 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.721913 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 15:33:35.722424 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.722407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 15:33:35.722642 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.722593 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 15:33:35.722886 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.722866 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.723076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723051 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2264\"" Jul 28 15:33:35.723149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723061 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.723149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723133 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.723247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.723393 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fqw5w\"" Jul 28 15:33:35.723564 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:35.723637 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.723614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.724654 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.723755 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:35.724654 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.723763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:35.725743 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.725725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.727140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.727123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.728435 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.728416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:35.728522 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.728475 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:35.730055 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.730036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.730179 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.730162 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 15:33:35.730459 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.730252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.732347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.732328 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6g7g2\"" Jul 28 15:33:35.732437 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.732351 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:33:35.732437 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.732405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 15:33:35.732627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.732612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2kwz4\"" Jul 28 15:33:35.743154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.743136 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 15:33:35.750662 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.750644 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:35.811737 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.811705 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 15:33:35.822770 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-netns\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-systemd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-modprobe-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-systemd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-run\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.822950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-netns\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswxq\" (UniqueName: \"kubernetes.io/projected/4e57260c-da09-4286-8a99-5739e4d5d6d5-kube-api-access-rswxq\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.822989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkbx\" (UniqueName: \"kubernetes.io/projected/79c18d1a-bf4d-4ced-987a-78745c891505-kube-api-access-9jkbx\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-socket-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-netns\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-lib-modules\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-netns\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-konnectivity-ca\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-cni-binary-copy\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-socket-dir-parent\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-daemon-config\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-config\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-socket-dir-parent\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-etc-tuned\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-kubelet-config\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-os-release\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-var-lib-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-tmp\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-os-release\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e57260c-da09-4286-8a99-5739e4d5d6d5-tmp-dir\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-dbus\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhxb\" (UniqueName: \"kubernetes.io/projected/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kube-api-access-rrhxb\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-agent-certs\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-kubelet\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e25eba0e-6332-4616-84e8-24ad2a645237-ovn-node-metrics-cert\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-var-lib-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.823857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c613a05c-0bc3-4bfa-ac92-7ff568727e95-host-slash\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-kubelet\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-system-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/5cc89da4-ef78-4830-9c8d-6e53ff954440-kube-api-access-n5fcq\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-konnectivity-ca\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-slash\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-cni-binary-copy\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-node-log\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c613a05c-0bc3-4bfa-ac92-7ff568727e95-host-slash\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823937 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-daemon-config\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-slash\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-config\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-sys-fs\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-env-overrides\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.824708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.823997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-system-cni-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-kubernetes\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-conf\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thx8d\" (UniqueName: \"kubernetes.io/projected/08026358-1001-4b5c-ad0c-7da3c7128309-kube-api-access-thx8d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824162 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-k8s-cni-cncf-io\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-multus-certs\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-node-log\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-sys\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-conf-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflp4\" (UniqueName: \"kubernetes.io/projected/e25eba0e-6332-4616-84e8-24ad2a645237-kube-api-access-bflp4\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-etc-selinux\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-serviceca\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-env-overrides\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-multus-certs\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.825550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-system-cni-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-multus-conf-dir\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-run-k8s-cni-cncf-io\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-os-release\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-etc-kubernetes\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-os-release\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-etc-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-system-cni-dir\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5xh\" (UniqueName: \"kubernetes.io/projected/4ee68990-196b-47a9-bf0a-397cc21718da-kube-api-access-gl5xh\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-serviceca\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-etc-kubernetes\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-multus\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-etc-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-log-socket\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-log-socket\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-multus\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.824966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ee68990-196b-47a9-bf0a-397cc21718da-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysconfig\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-var-lib-kubelet\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-bin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-hostroot\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-host-var-lib-cni-bin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-kubelet\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-hostroot\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-kubelet\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-netd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:35.826755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-netd\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-device-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6vm\" (UniqueName: \"kubernetes.io/projected/c613a05c-0bc3-4bfa-ac92-7ff568727e95-kube-api-access-8d6vm\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-cnibin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-systemd\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cc89da4-ef78-4830-9c8d-6e53ff954440-cnibin\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zqx\" (UniqueName: \"kubernetes.io/projected/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-kube-api-access-l8zqx\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-bin\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-script-lib\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e57260c-da09-4286-8a99-5739e4d5d6d5-hosts-file\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-host\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-host\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-host-cni-bin\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-cnibin\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-systemd-units\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-registration-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ee68990-196b-47a9-bf0a-397cc21718da-cnibin\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.827511 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c613a05c-0bc3-4bfa-ac92-7ff568727e95-iptables-alerter-script\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.825994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-ovn\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-host\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e25eba0e-6332-4616-84e8-24ad2a645237-ovnkube-script-lib\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-openvswitch\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-run-ovn\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e25eba0e-6332-4616-84e8-24ad2a645237-systemd-units\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.826462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c613a05c-0bc3-4bfa-ac92-7ff568727e95-iptables-alerter-script\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.828277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.828106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e25eba0e-6332-4616-84e8-24ad2a645237-ovn-node-metrics-cert\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.828582 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.828471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/252fea0f-b42f-4b9f-be6a-552ca4ce7fe6-agent-certs\") pod \"konnectivity-agent-fvkrk\" (UID: \"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6\") " pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:35.833360 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.833342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflp4\" (UniqueName: \"kubernetes.io/projected/e25eba0e-6332-4616-84e8-24ad2a645237-kube-api-access-bflp4\") pod \"ovnkube-node-7hv4d\" (UID: \"e25eba0e-6332-4616-84e8-24ad2a645237\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:35.833462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.833391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/5cc89da4-ef78-4830-9c8d-6e53ff954440-kube-api-access-n5fcq\") pod \"multus-xpdcx\" (UID: \"5cc89da4-ef78-4830-9c8d-6e53ff954440\") " pod="openshift-multus/multus-xpdcx" Jul 28 15:33:35.838982 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.838957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6vm\" (UniqueName: \"kubernetes.io/projected/c613a05c-0bc3-4bfa-ac92-7ff568727e95-kube-api-access-8d6vm\") pod \"iptables-alerter-rthgn\" (UID: \"c613a05c-0bc3-4bfa-ac92-7ff568727e95\") " pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:35.840299 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.840278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5xh\" (UniqueName: \"kubernetes.io/projected/4ee68990-196b-47a9-bf0a-397cc21718da-kube-api-access-gl5xh\") pod \"multus-additional-cni-plugins-vbs4j\" (UID: \"4ee68990-196b-47a9-bf0a-397cc21718da\") " pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:35.840821 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.840776 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zqx\" (UniqueName: \"kubernetes.io/projected/5968e5c7-27a6-4b49-b0af-d89ee40cdeab-kube-api-access-l8zqx\") pod \"node-ca-85bv6\" (UID: \"5968e5c7-27a6-4b49-b0af-d89ee40cdeab\") " pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:35.887812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.887769 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hk75s" Jul 28 15:33:35.897852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.897819 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hk75s" Jul 28 15:33:35.927280 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:35.927280 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-device-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-systemd\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e57260c-da09-4286-8a99-5739e4d5d6d5-hosts-file\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-registration-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-host\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-device-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-modprobe-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-run\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-systemd\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswxq\" (UniqueName: \"kubernetes.io/projected/4e57260c-da09-4286-8a99-5739e4d5d6d5-kube-api-access-rswxq\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkbx\" (UniqueName: \"kubernetes.io/projected/79c18d1a-bf4d-4ced-987a-78745c891505-kube-api-access-9jkbx\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-socket-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e57260c-da09-4286-8a99-5739e4d5d6d5-hosts-file\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-lib-modules\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-etc-tuned\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-kubelet-config\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-tmp\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-modprobe-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e57260c-da09-4286-8a99-5739e4d5d6d5-tmp-dir\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-host\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-dbus\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.927835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-run\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhxb\" (UniqueName: \"kubernetes.io/projected/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kube-api-access-rrhxb\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-registration-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.927914 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.927952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-socket-dir\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-lib-modules\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-kubelet-config\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.928261 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.428236792 +0000 UTC m=+3.109293810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e57260c-da09-4286-8a99-5739e4d5d6d5-tmp-dir\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.928424 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:35.928502 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.928483 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.428467627 +0000 UTC m=+3.109524625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-sys-fs\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-kubernetes\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-conf\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-kubernetes\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysctl-conf\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-sys-fs\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thx8d\" (UniqueName: \"kubernetes.io/projected/08026358-1001-4b5c-ad0c-7da3c7128309-kube-api-access-thx8d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29e3fd0b-5634-4368-a41a-2fa9768aa695-dbus\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-sys\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.928969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-etc-selinux\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysconfig\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-var-lib-kubelet\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-sys\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-var-lib-kubelet\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.929649 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01ca9e4c-56f6-4e88-85f3-b7451cc38319-etc-selinux\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.929649 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.929152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08026358-1001-4b5c-ad0c-7da3c7128309-etc-sysconfig\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.930469 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.930431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-etc-tuned\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.930605 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.930589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08026358-1001-4b5c-ad0c-7da3c7128309-tmp\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.936350 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.936219 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:35.936350 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.936245 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:35.936350 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.936257 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:35.936350 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:35.936310 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:36.436292474 +0000 UTC m=+3.117349481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:35.938215 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.938194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thx8d\" (UniqueName: \"kubernetes.io/projected/08026358-1001-4b5c-ad0c-7da3c7128309-kube-api-access-thx8d\") pod \"tuned-qmn8h\" (UID: \"08026358-1001-4b5c-ad0c-7da3c7128309\") " pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:35.938303 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.938202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhxb\" (UniqueName: \"kubernetes.io/projected/01ca9e4c-56f6-4e88-85f3-b7451cc38319-kube-api-access-rrhxb\") pod \"aws-ebs-csi-driver-node-4mngs\" (UID: \"01ca9e4c-56f6-4e88-85f3-b7451cc38319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:35.938956 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.938935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswxq\" (UniqueName: \"kubernetes.io/projected/4e57260c-da09-4286-8a99-5739e4d5d6d5-kube-api-access-rswxq\") pod \"node-resolver-xz94w\" (UID: \"4e57260c-da09-4286-8a99-5739e4d5d6d5\") " pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:35.939067 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:35.938993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkbx\" (UniqueName: \"kubernetes.io/projected/79c18d1a-bf4d-4ced-987a-78745c891505-kube-api-access-9jkbx\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:36.029352 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.029273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:36.035140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.035120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85bv6" Jul 28 15:33:36.045476 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.045453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rthgn" Jul 28 15:33:36.051687 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.051668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" Jul 28 15:33:36.059586 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.059566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xpdcx" Jul 28 15:33:36.068269 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.068250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:33:36.075086 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.075062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xz94w" Jul 28 15:33:36.083848 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.083830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" Jul 28 15:33:36.087338 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.087317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" Jul 28 15:33:36.432835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.432735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:36.432835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.432783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:36.433054 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.432918 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:36.433054 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.432919 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:36.433054 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.432982 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.432963296 +0000 UTC m=+4.114020286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:36.433054 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.433002 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.432993209 +0000 UTC m=+4.114050203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:36.533652 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.533617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:36.533860 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.533837 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:36.533929 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.533871 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:36.533929 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.533885 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:36.534009 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:36.533959 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:37.533939343 +0000 UTC m=+4.214996354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:36.648898 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.648868 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee68990_196b_47a9_bf0a_397cc21718da.slice/crio-601b925bb2e517e8a15cf16081e2b0d83d063eebffbfb0002e6a04e9eea9a97a WatchSource:0}: Error finding container 601b925bb2e517e8a15cf16081e2b0d83d063eebffbfb0002e6a04e9eea9a97a: Status 404 returned error can't find the container with id 601b925bb2e517e8a15cf16081e2b0d83d063eebffbfb0002e6a04e9eea9a97a Jul 28 15:33:36.649647 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.649547 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5968e5c7_27a6_4b49_b0af_d89ee40cdeab.slice/crio-3b212dcd8f65253bb6b800dddcaaff1b09154d5f670447a8b33cd21e972cf6f6 WatchSource:0}: Error finding container 3b212dcd8f65253bb6b800dddcaaff1b09154d5f670447a8b33cd21e972cf6f6: Status 404 returned error can't find the container with id 3b212dcd8f65253bb6b800dddcaaff1b09154d5f670447a8b33cd21e972cf6f6 Jul 28 15:33:36.650749 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.650731 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc89da4_ef78_4830_9c8d_6e53ff954440.slice/crio-eaf7af671b46c7322fa78f84b1b914649ec9beb22c5100113eead90c197118a9 WatchSource:0}: Error finding container eaf7af671b46c7322fa78f84b1b914649ec9beb22c5100113eead90c197118a9: Status 404 returned error can't find the container with id eaf7af671b46c7322fa78f84b1b914649ec9beb22c5100113eead90c197118a9 Jul 28 15:33:36.652331 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.652314 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252fea0f_b42f_4b9f_be6a_552ca4ce7fe6.slice/crio-71ad51da154d7f92826941e17e96648f1f724b83382e656341630384f264c7b6 WatchSource:0}: Error finding container 71ad51da154d7f92826941e17e96648f1f724b83382e656341630384f264c7b6: Status 404 returned error can't find the container with id 71ad51da154d7f92826941e17e96648f1f724b83382e656341630384f264c7b6 Jul 28 15:33:36.653547 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.653523 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ca9e4c_56f6_4e88_85f3_b7451cc38319.slice/crio-6407acbb5a6f0fc00e1fd9be1fcd85f9a8476bf6554b18680eb29151aa9f5848 WatchSource:0}: Error finding container 6407acbb5a6f0fc00e1fd9be1fcd85f9a8476bf6554b18680eb29151aa9f5848: Status 404 returned error can't find the container with id 6407acbb5a6f0fc00e1fd9be1fcd85f9a8476bf6554b18680eb29151aa9f5848 Jul 28 15:33:36.657537 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.657419 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08026358_1001_4b5c_ad0c_7da3c7128309.slice/crio-3d587e916b37dc7d7dacaf4fdffaf58fe9e40df8c002bd6c80dd2f63f7575d8d WatchSource:0}: Error finding container 3d587e916b37dc7d7dacaf4fdffaf58fe9e40df8c002bd6c80dd2f63f7575d8d: Status 404 returned error can't find the container with id 3d587e916b37dc7d7dacaf4fdffaf58fe9e40df8c002bd6c80dd2f63f7575d8d Jul 28 15:33:36.659016 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.658996 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e57260c_da09_4286_8a99_5739e4d5d6d5.slice/crio-720ffe6f09a270abd54e87225489809e8c5fc6a13f4725b5af247d6f1a563c47 WatchSource:0}: Error finding container 720ffe6f09a270abd54e87225489809e8c5fc6a13f4725b5af247d6f1a563c47: Status 404 returned error can't find the container with id 720ffe6f09a270abd54e87225489809e8c5fc6a13f4725b5af247d6f1a563c47 Jul 28 15:33:36.662426 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:33:36.661392 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25eba0e_6332_4616_84e8_24ad2a645237.slice/crio-bb196fc53e963cec62425aa476380823334eb28563c7f8dfe87931404f602fa6 WatchSource:0}: Error finding container bb196fc53e963cec62425aa476380823334eb28563c7f8dfe87931404f602fa6: Status 404 returned error can't find the container with id bb196fc53e963cec62425aa476380823334eb28563c7f8dfe87931404f602fa6 Jul 28 15:33:36.860122 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.859923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" event={"ID":"08026358-1001-4b5c-ad0c-7da3c7128309","Type":"ContainerStarted","Data":"3d587e916b37dc7d7dacaf4fdffaf58fe9e40df8c002bd6c80dd2f63f7575d8d"} Jul 28 15:33:36.861062 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.861038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" event={"ID":"01ca9e4c-56f6-4e88-85f3-b7451cc38319","Type":"ContainerStarted","Data":"6407acbb5a6f0fc00e1fd9be1fcd85f9a8476bf6554b18680eb29151aa9f5848"} Jul 28 15:33:36.862014 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.861992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fvkrk" event={"ID":"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6","Type":"ContainerStarted","Data":"71ad51da154d7f92826941e17e96648f1f724b83382e656341630384f264c7b6"} Jul 28 15:33:36.862989 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.862968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xpdcx" event={"ID":"5cc89da4-ef78-4830-9c8d-6e53ff954440","Type":"ContainerStarted","Data":"eaf7af671b46c7322fa78f84b1b914649ec9beb22c5100113eead90c197118a9"} Jul 28 15:33:36.864216 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.864196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85bv6" event={"ID":"5968e5c7-27a6-4b49-b0af-d89ee40cdeab","Type":"ContainerStarted","Data":"3b212dcd8f65253bb6b800dddcaaff1b09154d5f670447a8b33cd21e972cf6f6"} Jul 28 15:33:36.865235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.865213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rthgn" event={"ID":"c613a05c-0bc3-4bfa-ac92-7ff568727e95","Type":"ContainerStarted","Data":"0e7d094fdb75a1b83f1476b87d8c771753edc287d723b3cfccb912b88869a4c4"} Jul 28 15:33:36.866250 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.866224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerStarted","Data":"601b925bb2e517e8a15cf16081e2b0d83d063eebffbfb0002e6a04e9eea9a97a"} Jul 28 15:33:36.867947 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.867929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" event={"ID":"372e08fc4324f245307306950cbd07f2","Type":"ContainerStarted","Data":"dd43e29347341c8d0f7e8d564bf5cc0381ca98e5cc9a10de94b712d3ad673271"} Jul 28 15:33:36.869002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.868984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"bb196fc53e963cec62425aa476380823334eb28563c7f8dfe87931404f602fa6"} Jul 28 15:33:36.869927 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.869910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xz94w" event={"ID":"4e57260c-da09-4286-8a99-5739e4d5d6d5","Type":"ContainerStarted","Data":"720ffe6f09a270abd54e87225489809e8c5fc6a13f4725b5af247d6f1a563c47"} Jul 28 15:33:36.900962 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.900930 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:35 +0000 UTC" deadline="2027-12-25 00:48:58.410807954 +0000 UTC" Jul 28 15:33:36.900962 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:36.900961 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12345h15m21.509850327s" Jul 28 15:33:37.440737 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.440701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:37.440946 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.440758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:37.440946 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.440901 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:37.441058 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.440967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.440949297 +0000 UTC m=+6.122006301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:37.441541 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.441343 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:37.441541 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.441393 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.441376272 +0000 UTC m=+6.122433269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:37.541448 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.541409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:37.541653 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.541636 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:37.541738 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.541661 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:37.541738 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.541673 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:37.541738 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.541734 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.541714797 +0000 UTC m=+6.222771789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:37.832595 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.832514 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.853660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.853790 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.854225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.854335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.854436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:37.854553 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:37.854517 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:37.901784 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.901739 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:35 +0000 UTC" deadline="2028-04-21 18:01:04.793104228 +0000 UTC" Jul 28 15:33:37.901784 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.901782 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15194h27m26.891326252s" Jul 28 15:33:37.915783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.915710 2580 generic.go:356] "Generic (PLEG): container finished" podID="acfc96076c26aaf99e282a8d26fbce05" containerID="ff2bc09700f63a77249eb9b6187895f7b5e52c472fc85ce35a5079b246547b60" exitCode=0 Jul 28 15:33:37.915968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.915852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"acfc96076c26aaf99e282a8d26fbce05","Type":"ContainerDied","Data":"ff2bc09700f63a77249eb9b6187895f7b5e52c472fc85ce35a5079b246547b60"} Jul 28 15:33:37.938138 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:37.937864 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" podStartSLOduration=3.937842363 podStartE2EDuration="3.937842363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:36.88846974 +0000 UTC m=+3.569526752" watchObservedRunningTime="2026-07-28 15:33:37.937842363 +0000 UTC m=+4.618899373" Jul 28 15:33:38.944096 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:38.943275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"acfc96076c26aaf99e282a8d26fbce05","Type":"ContainerStarted","Data":"650f742b044c7c3fff3a384a3d50f6c63e0af80f5bbd7b277cd839ec33e64e63"} Jul 28 15:33:38.964413 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:38.964036 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" podStartSLOduration=4.964013104 podStartE2EDuration="4.964013104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:38.96323934 +0000 UTC m=+5.644296364" watchObservedRunningTime="2026-07-28 15:33:38.964013104 +0000 UTC m=+5.645070117" Jul 28 15:33:39.457837 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.457638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:39.457837 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.457697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:39.458077 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.457844 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:39.458077 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.457906 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.457888572 +0000 UTC m=+10.138945567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:39.458726 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.458316 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:39.458726 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.458366 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.458351184 +0000 UTC m=+10.139408179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:39.558754 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.558712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:39.558987 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.558956 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:39.558987 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.558984 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:39.559095 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.558996 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:39.559095 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.559055 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.559036498 +0000 UTC m=+10.240093510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.855018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.855040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:39.855029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.855149 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.855286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:39.855516 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:39.855369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:41.853420 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:41.853346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:41.853870 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:41.853359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:41.853870 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:41.853463 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:41.853870 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:41.853482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:41.853870 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:41.853566 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:41.853870 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:41.853623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:43.494863 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.494826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:43.495427 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.494877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:43.495427 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.494988 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:43.495427 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.495038 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.49502002 +0000 UTC m=+18.176077010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:43.495427 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.495396 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:43.495678 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.495443 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.495427325 +0000 UTC m=+18.176484318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:43.595882 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.595128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:43.595882 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.595410 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:43.595882 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.595433 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:43.595882 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.595497 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:43.595882 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.595542 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:51.595528851 +0000 UTC m=+18.276585841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:43.853449 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.853377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:43.853601 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.853465 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:43.853852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.853824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:43.853952 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.853932 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:43.854138 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:43.854030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:43.854138 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:43.854106 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:45.862921 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:45.862878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:45.863597 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:45.862879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:45.863597 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:45.863130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:45.863597 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:45.863181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:45.863597 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:45.863297 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:45.863597 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:45.863421 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:47.852828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:47.852709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:47.852828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:47.852755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:47.852828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:47.852720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:47.853376 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:47.852892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:47.853376 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:47.853049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:47.853376 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:47.853122 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:49.852904 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:49.852858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:49.853311 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:49.852858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:49.853311 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:49.853007 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:49.853311 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:49.852869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:49.853311 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:49.853077 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:49.853311 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:49.853145 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:51.556061 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.556028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:51.556061 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.556072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:51.556521 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.556186 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:51.556521 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.556186 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:51.556521 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.556249 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:07.556230251 +0000 UTC m=+34.237287240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:33:51.556521 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.556270 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:07.556257988 +0000 UTC m=+34.237314979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:33:51.657187 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.657125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:51.657399 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.657307 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:33:51.657399 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.657331 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:33:51.657399 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.657343 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:51.657529 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.657407 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:07.657389419 +0000 UTC m=+34.338446422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:33:51.855690 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.855615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:51.855690 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.855654 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:51.855899 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.855731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:51.855899 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.855770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:51.855899 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:51.855840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:51.856014 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:51.855950 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:53.855459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:53.855478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:53.855579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:53.855577 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:53.855674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:53.855823 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:53.855765 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:55.856854 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.856588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:55.857460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.856622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:55.857460 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:55.856967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:55.857460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.856628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:55.857460 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:55.857028 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:55.857460 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:55.857070 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:55.981218 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981084 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:33:55.981708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981688 2580 generic.go:356] "Generic (PLEG): container finished" podID="e25eba0e-6332-4616-84e8-24ad2a645237" containerID="898bc93ab594800f12a117b6471645b45549c1958329cebdf242a1f4d1637bfc" exitCode=1 Jul 28 15:33:55.981760 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"1df155c491beacee4fac1e53b38a73432e52121df7350e83a5ec26665854020f"} Jul 28 15:33:55.981868 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"4eaf9a538ba716685796531cb461b6a4c2156e1f67a9553e297a207e4b456ca1"} Jul 28 15:33:55.981925 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerDied","Data":"898bc93ab594800f12a117b6471645b45549c1958329cebdf242a1f4d1637bfc"} Jul 28 15:33:55.981998 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.981987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"b51fff9c9c5df1951a2b0f293b24a4d5e9237af9c44090f0f0ce1714dfccf6fd"} Jul 28 15:33:55.983253 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.983230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xz94w" event={"ID":"4e57260c-da09-4286-8a99-5739e4d5d6d5","Type":"ContainerStarted","Data":"a8bba216554c44d29bf54fb7840cada49fa211b81c8ad357f7900cdbd67c8250"} Jul 28 15:33:55.986667 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.985627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" event={"ID":"08026358-1001-4b5c-ad0c-7da3c7128309","Type":"ContainerStarted","Data":"2b77c10f19a00ee1854d533b62b82c95f7475674adbdb76ab628ac6239ba3aba"} Jul 28 15:33:55.987754 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.987731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" event={"ID":"01ca9e4c-56f6-4e88-85f3-b7451cc38319","Type":"ContainerStarted","Data":"4ca52511a7598185bf51b04243efa42201bc32162265d52f28b53a7e8fbbe672"} Jul 28 15:33:55.989314 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.989291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fvkrk" event={"ID":"252fea0f-b42f-4b9f-be6a-552ca4ce7fe6","Type":"ContainerStarted","Data":"1611010261ac171b5524ba24504add67d99d25ccd6345e2b987c219e60c2360f"} Jul 28 15:33:55.990974 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.990945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xpdcx" event={"ID":"5cc89da4-ef78-4830-9c8d-6e53ff954440","Type":"ContainerStarted","Data":"59ea2799ae341acf216939c9c45f148ded64323f726d3a683f612edd410e5991"} Jul 28 15:33:55.992428 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.992405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85bv6" event={"ID":"5968e5c7-27a6-4b49-b0af-d89ee40cdeab","Type":"ContainerStarted","Data":"275a2c1e7974f21cdcc0a23b561dcbbb5f7eb84d9314788703de1402397abe34"} Jul 28 15:33:55.994112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.994087 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="4080d95d0a32b014e38ac3b7a98494d392bbe98e58bd62544946233463ac6c6f" exitCode=0 Jul 28 15:33:55.994205 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:55.994123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"4080d95d0a32b014e38ac3b7a98494d392bbe98e58bd62544946233463ac6c6f"} Jul 28 15:33:56.002093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.002055 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xz94w" podStartSLOduration=3.338289191 podStartE2EDuration="22.002030751s" totalImagesPullingTime="18.66374156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.001372212 +0000 UTC m=+22.682429225" watchObservedRunningTime="2026-07-28 15:33:56.002030751 +0000 UTC m=+22.683087764" Jul 28 15:33:56.018090 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.018049 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fvkrk" podStartSLOduration=8.719022623 podStartE2EDuration="23.018034851s" totalImagesPullingTime="14.299012228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.017334209 +0000 UTC m=+22.698391222" watchObservedRunningTime="2026-07-28 15:33:56.018034851 +0000 UTC m=+22.699091862" Jul 28 15:33:56.065429 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.065375 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xpdcx" podStartSLOduration=4.369539198 podStartE2EDuration="23.065350097s" totalImagesPullingTime="18.695810899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.064647679 +0000 UTC m=+22.745704706" watchObservedRunningTime="2026-07-28 15:33:56.065350097 +0000 UTC m=+22.746407110" Jul 28 15:33:56.082863 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.082816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-85bv6" podStartSLOduration=4.42721835 podStartE2EDuration="23.08278402s" totalImagesPullingTime="18.65556567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.082547406 +0000 UTC m=+22.763604426" watchObservedRunningTime="2026-07-28 15:33:56.08278402 +0000 UTC m=+22.763841033" Jul 28 15:33:56.109274 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.109221 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qmn8h" podStartSLOduration=3.44072125 podStartE2EDuration="22.109205386s" totalImagesPullingTime="18.668484136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:56.10876971 +0000 UTC m=+22.789826733" watchObservedRunningTime="2026-07-28 15:33:56.109205386 +0000 UTC m=+22.790262399" Jul 28 15:33:56.295398 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.295272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:56.299413 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.299387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:56.704568 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.704540 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 15:33:56.781509 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.781395 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:33:56.704560459Z","UUID":"4e9125c6-8d34-4816-b9ad-6e5af809555a","Handler":null,"Name":"","Endpoint":""} Jul 28 15:33:56.783053 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.783031 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 15:33:56.783177 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:56.783069 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 15:33:57.001357 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.001326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" event={"ID":"01ca9e4c-56f6-4e88-85f3-b7451cc38319","Type":"ContainerStarted","Data":"3079ddff9d2e5d9a9f778753e6877889fa0fc22c097967de12e99b85b213d69e"} Jul 28 15:33:57.002923 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.002894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rthgn" event={"ID":"c613a05c-0bc3-4bfa-ac92-7ff568727e95","Type":"ContainerStarted","Data":"9dede45f4620e2d15bee12bd4b8a9fbfe9c2d5a705af153179e12bacd15ed92e"} Jul 28 15:33:57.006469 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.006450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:33:57.007146 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.007112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"360d74f401e30072b55957e548f648add858b4a3261ea45f9475151d6250ee7e"} Jul 28 15:33:57.007247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.007212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"6da89e279c01261825798cf0171e94f11f4e21cddf5f84bfe8232765be43dbee"} Jul 28 15:33:57.008010 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.007992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:57.008166 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.008039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fvkrk" Jul 28 15:33:57.029620 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.029555 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rthgn" podStartSLOduration=5.388861171 podStartE2EDuration="24.029536634s" totalImagesPullingTime="18.640675463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:57.028551833 +0000 UTC m=+23.709608836" watchObservedRunningTime="2026-07-28 15:33:57.029536634 +0000 UTC m=+23.710593650" Jul 28 15:33:57.852863 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.852828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:57.853024 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.852828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:57.853024 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:57.852963 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:57.853024 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:57.852841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:57.853187 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:57.853022 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:33:57.853187 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:57.853078 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:58.011230 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:58.011194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" event={"ID":"01ca9e4c-56f6-4e88-85f3-b7451cc38319","Type":"ContainerStarted","Data":"6e74fdfcbbfcc25ef83a14364aa841cc4f3eac1aee3c8287ab75d16631f477eb"} Jul 28 15:33:58.034973 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:58.034919 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4mngs" podStartSLOduration=3.437796641 podStartE2EDuration="24.034903414s" totalImagesPullingTime="20.597106773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:58.034152009 +0000 UTC m=+24.715209025" watchObservedRunningTime="2026-07-28 15:33:58.034903414 +0000 UTC m=+24.715960426" Jul 28 15:33:59.016987 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:59.016953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:33:59.017420 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:59.017382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"bfc31eb66f0f37d89631ccbdd605c468b98ce21c0d088b75806cbf8ffb59bccb"} Jul 28 15:33:59.852932 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:59.852726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:33:59.853120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:59.852726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:33:59.853120 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:59.853037 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:33:59.853120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:33:59.852736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:33:59.853120 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:59.853100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:33:59.853329 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:33:59.853169 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:01.025355 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.025329 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:34:01.026196 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.025724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"d5927bf941b75fe1068ee64ec04ed0689627092dc5b37cc02f1aae350f6e56d5"} Jul 28 15:34:01.026196 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.026000 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:01.026347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.026197 2580 scope.go:117] "RemoveContainer" containerID="898bc93ab594800f12a117b6471645b45549c1958329cebdf242a1f4d1637bfc" Jul 28 15:34:01.027632 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.027609 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="993de7b9c554fb73e1b409637d50689ab4b211c63f794c442c265b6c473c4139" exitCode=0 Jul 28 15:34:01.027717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.027640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"993de7b9c554fb73e1b409637d50689ab4b211c63f794c442c265b6c473c4139"} Jul 28 15:34:01.043055 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.043033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:01.852727 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.852572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:01.852844 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.852627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:01.852905 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:01.852835 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:34:01.852905 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:01.852687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:01.853015 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:01.852906 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:01.853069 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:01.853003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:34:02.032418 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.032385 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="5ba91226a4a157df34ee72385411be0b39c8353b0406c69400a89b3a4574ded7" exitCode=0 Jul 28 15:34:02.032849 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.032467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"5ba91226a4a157df34ee72385411be0b39c8353b0406c69400a89b3a4574ded7"} Jul 28 15:34:02.036617 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.036590 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:34:02.036952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.036930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" event={"ID":"e25eba0e-6332-4616-84e8-24ad2a645237","Type":"ContainerStarted","Data":"94df9d569a95daf07d8fd7550289db95755dc7611c0410f963e683e3f319b537"} Jul 28 15:34:02.037176 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.037148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:02.037293 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.037207 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:02.052583 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.052560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:02.090703 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.090606 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" podStartSLOduration=9.370910175 podStartE2EDuration="28.090591208s" totalImagesPullingTime="18.719681033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.090329306 +0000 UTC m=+28.771386319" watchObservedRunningTime="2026-07-28 15:34:02.090591208 +0000 UTC m=+28.771648235" Jul 28 15:34:02.576496 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.576423 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gs4v8"] Jul 28 15:34:02.576692 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.576569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:02.576749 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:02.576694 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:34:02.577844 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.577818 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxplb"] Jul 28 15:34:02.577950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.577908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:02.578024 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:02.578004 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:02.578289 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.578273 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbcgv"] Jul 28 15:34:02.578350 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:02.578338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:02.578441 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:02.578424 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:34:03.040997 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:03.040956 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="77590e281fc8c0c26e92cce219005ba42b21474074fc1bca2434ad59110d7797" exitCode=0 Jul 28 15:34:03.041457 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:03.041007 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"77590e281fc8c0c26e92cce219005ba42b21474074fc1bca2434ad59110d7797"} Jul 28 15:34:03.854367 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:03.854323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:03.854541 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:03.854444 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:34:03.854541 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:03.854487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:03.854635 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:03.854562 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:04.852756 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:04.852690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:04.853174 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:04.852853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:34:05.853492 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:05.853282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:05.853903 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:05.853583 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:34:05.853971 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:05.853310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:05.854072 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:05.854047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:06.853077 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:06.852991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:06.853276 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:06.853134 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gs4v8" podUID="79c18d1a-bf4d-4ced-987a-78745c891505" Jul 28 15:34:07.575136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:07.575112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:07.575582 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:07.575148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:07.575582 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.575249 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 15:34:07.575582 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.575256 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:34:07.575582 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.575296 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret podName:29e3fd0b-5634-4368-a41a-2fa9768aa695 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:39.575283953 +0000 UTC m=+66.256340947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret") pod "global-pull-secret-syncer-cbcgv" (UID: "29e3fd0b-5634-4368-a41a-2fa9768aa695") : object "kube-system"/"original-pull-secret" not registered Jul 28 15:34:07.575582 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.575310 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs podName:79c18d1a-bf4d-4ced-987a-78745c891505 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:39.575304329 +0000 UTC m=+66.256361318 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs") pod "network-metrics-daemon-gs4v8" (UID: "79c18d1a-bf4d-4ced-987a-78745c891505") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 15:34:07.676274 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:07.676203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:07.676412 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.676341 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 15:34:07.676412 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.676355 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 15:34:07.676412 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.676363 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2xtq2 for pod openshift-network-diagnostics/network-check-target-hxplb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:34:07.676412 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.676410 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2 podName:727244f6-be9a-4e62-8294-29ef4c4d7936 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:39.67639739 +0000 UTC m=+66.357454380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xtq2" (UniqueName: "kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2") pod "network-check-target-hxplb" (UID: "727244f6-be9a-4e62-8294-29ef4c4d7936") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 15:34:07.852844 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:07.852787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:07.853004 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:07.852854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:07.853004 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.852932 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbcgv" podUID="29e3fd0b-5634-4368-a41a-2fa9768aa695" Jul 28 15:34:07.853074 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:07.853042 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxplb" podUID="727244f6-be9a-4e62-8294-29ef4c4d7936" Jul 28 15:34:08.056642 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.056565 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="be16e612b362dc4566adeb39d810ec36814e430bd623e53bbc441a1213845c75" exitCode=0 Jul 28 15:34:08.056642 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.056615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"be16e612b362dc4566adeb39d810ec36814e430bd623e53bbc441a1213845c75"} Jul 28 15:34:08.100886 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.100865 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeReady" Jul 28 15:34:08.101008 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.101000 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 15:34:08.141071 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.141042 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:34:08.157739 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.157717 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57cdcfc679-kw468"] Jul 28 15:34:08.157864 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.157835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.160846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.160790 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxdc4\"" Jul 28 15:34:08.161462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.161439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 15:34:08.161555 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.161528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 15:34:08.170214 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.169270 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 15:34:08.170214 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.170058 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b"] Jul 28 15:34:08.171210 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.171184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.174786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.175400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.175519 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.175637 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.175760 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.176118 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.175868 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jpm9g\"" Jul 28 15:34:08.176467 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.176207 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 15:34:08.179298 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.179110 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 15:34:08.179590 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.179569 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw"] Jul 28 15:34:08.180792 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.179771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" Jul 28 15:34:08.182607 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.182549 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.187559 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.187127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.187660 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.187568 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qb4cr\"" Jul 28 15:34:08.191353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.191321 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-s6mqj"] Jul 28 15:34:08.191594 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.191578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.195185 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.195164 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.195284 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.195165 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l6dkt\"" Jul 28 15:34:08.195284 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.195203 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.195284 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.195214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 15:34:08.199714 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.199696 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp"] Jul 28 15:34:08.199876 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.199852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.202372 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.202355 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5gx9n\"" Jul 28 15:34:08.202465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.202386 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.202530 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.202491 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 15:34:08.202624 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.202607 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 15:34:08.202690 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.202619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.207908 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.207890 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 15:34:08.207990 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.207964 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4"] Jul 28 15:34:08.208119 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.208103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.212197 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.212172 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 15:34:08.212610 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.212595 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 15:34:08.212883 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.212848 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.213136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.213119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.213226 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.213175 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5rqzv\"" Jul 28 15:34:08.216305 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.216287 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9"] Jul 28 15:34:08.216443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.216428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.219093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.219074 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 15:34:08.219200 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.219096 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 15:34:08.219200 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.219165 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.219301 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.219213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.219378 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.219364 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xnc64\"" Jul 28 15:34:08.221899 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.221882 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgx6t"] Jul 28 15:34:08.221984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.221975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.224323 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.224308 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.224887 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.224872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 15:34:08.224945 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.224911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.224997 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.224951 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-t8vd6\"" Jul 28 15:34:08.224997 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.224986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 15:34:08.229988 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.229971 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5mtj5"] Jul 28 15:34:08.230088 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.230075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.232912 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.232897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.233050 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.233039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 15:34:08.233130 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.233110 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 15:34:08.233194 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.233141 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.233239 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.233191 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6jrd9\"" Jul 28 15:34:08.237982 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.237966 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 15:34:08.238212 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.238198 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf"] Jul 28 15:34:08.238304 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.238291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.240713 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.240694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sv4z4\"" Jul 28 15:34:08.240814 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.240727 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 15:34:08.240935 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.240922 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 15:34:08.247166 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.247149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:34:08.247255 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.247174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j5ftk"] Jul 28 15:34:08.247316 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.247260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" Jul 28 15:34:08.249918 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.249902 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.249979 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.249920 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.250153 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.250140 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v7gts\"" Jul 28 15:34:08.254576 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b"] Jul 28 15:34:08.254638 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4"] Jul 28 15:34:08.254638 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254600 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp"] Jul 28 15:34:08.254638 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254618 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgx6t"] Jul 28 15:34:08.254638 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw"] Jul 28 15:34:08.254749 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9"] Jul 28 15:34:08.254749 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.254867 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254675 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57cdcfc679-kw468"] Jul 28 15:34:08.254867 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254824 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf"] Jul 28 15:34:08.254867 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j5ftk"] Jul 28 15:34:08.254867 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254866 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5mtj5"] Jul 28 15:34:08.255006 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.254882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-s6mqj"] Jul 28 15:34:08.257445 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.257429 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qlvzt\"" Jul 28 15:34:08.257445 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.257445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 15:34:08.257568 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.257459 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 15:34:08.257623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.257614 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 15:34:08.280410 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.280508 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5nl\" (UniqueName: \"kubernetes.io/projected/891844af-0fb5-4513-b06b-4c23d070cc2e-kube-api-access-fv5nl\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.280508 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.280508 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.280608 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7c9c\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.280608 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-default-certificate\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.280608 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmrq\" (UniqueName: \"kubernetes.io/projected/2579dcd3-2e50-45eb-bef3-de71d89fcfb9-kube-api-access-bqmrq\") pod \"volume-data-source-validator-549c6d54cf-h5h2b\" (UID: \"2579dcd3-2e50-45eb-bef3-de71d89fcfb9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" Jul 28 15:34:08.280698 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rcc\" (UniqueName: \"kubernetes.io/projected/61375885-2520-41d7-a030-394abf63bba1-kube-api-access-k5rcc\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.280698 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-trusted-ca\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.280698 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e8143238-a595-4467-81b2-0ee06280cd4d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.280698 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.280698 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68j5\" (UniqueName: \"kubernetes.io/projected/8975da34-3728-4e3d-8f6a-e782583c3e94-kube-api-access-h68j5\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltzz\" (UniqueName: \"kubernetes.io/projected/e8143238-a595-4467-81b2-0ee06280cd4d-kube-api-access-9ltzz\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-stats-auth\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891844af-0fb5-4513-b06b-4c23d070cc2e-serving-cert\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.280872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-config\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.281069 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.281069 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.281069 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.281069 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.280959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.382818 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5nl\" (UniqueName: \"kubernetes.io/projected/891844af-0fb5-4513-b06b-4c23d070cc2e-kube-api-access-fv5nl\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68j5\" (UniqueName: \"kubernetes.io/projected/8975da34-3728-4e3d-8f6a-e782583c3e94-kube-api-access-h68j5\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.382998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-serving-cert\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.383025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7c9c\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmrq\" (UniqueName: \"kubernetes.io/projected/2579dcd3-2e50-45eb-bef3-de71d89fcfb9-kube-api-access-bqmrq\") pod \"volume-data-source-validator-549c6d54cf-h5h2b\" (UID: \"2579dcd3-2e50-45eb-bef3-de71d89fcfb9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-snapshots\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltzz\" (UniqueName: \"kubernetes.io/projected/e8143238-a595-4467-81b2-0ee06280cd4d-kube-api-access-9ltzz\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-stats-auth\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7n7\" (UniqueName: \"kubernetes.io/projected/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-kube-api-access-gm7n7\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjs85\" (UniqueName: \"kubernetes.io/projected/832386bd-d9d4-477c-a341-d7187c9a08e5-kube-api-access-kjs85\") pod \"network-check-source-7d88cb55d8-rtpzf\" (UID: \"832386bd-d9d4-477c-a341-d7187c9a08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djcr\" (UniqueName: \"kubernetes.io/projected/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-kube-api-access-4djcr\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66dab572-5fdd-4a86-ba23-8b43f447b27a-cert\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-tmp\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-config\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.383446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvhd\" (UniqueName: \"kubernetes.io/projected/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-kube-api-access-6qvhd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-tmp-dir\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-default-certificate\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rcc\" (UniqueName: \"kubernetes.io/projected/61375885-2520-41d7-a030-394abf63bba1-kube-api-access-k5rcc\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-trusted-ca\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-config-volume\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e8143238-a595-4467-81b2-0ee06280cd4d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcd2\" (UniqueName: \"kubernetes.io/projected/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-kube-api-access-jmcd2\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mct\" (UniqueName: \"kubernetes.io/projected/66dab572-5fdd-4a86-ba23-8b43f447b27a-kube-api-access-m4mct\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.384156 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891844af-0fb5-4513-b06b-4c23d070cc2e-serving-cert\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.384839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.384839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.383955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-config\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.384839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.384700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-config\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.386648 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386569 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:08.386756 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386651 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:08.386756 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386661 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:08.386756 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386719 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.8866998 +0000 UTC m=+35.567756790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:08.386756 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386740 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.886729812 +0000 UTC m=+35.567786806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:08.386940 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386869 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:08.386940 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.386937 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.886916049 +0000 UTC m=+35.567973053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:08.387183 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.387157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.387455 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.387184 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.887157038 +0000 UTC m=+35.568214046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:08.388062 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.388044 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:08.388693 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.388642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e8143238-a595-4467-81b2-0ee06280cd4d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.388786 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.388723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/891844af-0fb5-4513-b06b-4c23d070cc2e-trusted-ca\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.388901 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.388877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.389100 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.389078 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.889058246 +0000 UTC m=+35.570115241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:08.389228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.389162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.390148 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.390123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891844af-0fb5-4513-b06b-4c23d070cc2e-serving-cert\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.390231 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.390133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.390231 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.390166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.390357 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.390332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-default-certificate\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.390653 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.390635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-stats-auth\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.399304 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.399278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68j5\" (UniqueName: \"kubernetes.io/projected/8975da34-3728-4e3d-8f6a-e782583c3e94-kube-api-access-h68j5\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.399749 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.399729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rcc\" (UniqueName: \"kubernetes.io/projected/61375885-2520-41d7-a030-394abf63bba1-kube-api-access-k5rcc\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.399895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.399882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7c9c\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.400839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.400816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5nl\" (UniqueName: \"kubernetes.io/projected/891844af-0fb5-4513-b06b-4c23d070cc2e-kube-api-access-fv5nl\") pod \"console-operator-584dcb6b54-s6mqj\" (UID: \"891844af-0fb5-4513-b06b-4c23d070cc2e\") " pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.400961 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.400946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltzz\" (UniqueName: \"kubernetes.io/projected/e8143238-a595-4467-81b2-0ee06280cd4d-kube-api-access-9ltzz\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.401005 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.400953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.410154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.410134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmrq\" (UniqueName: \"kubernetes.io/projected/2579dcd3-2e50-45eb-bef3-de71d89fcfb9-kube-api-access-bqmrq\") pod \"volume-data-source-validator-549c6d54cf-h5h2b\" (UID: \"2579dcd3-2e50-45eb-bef3-de71d89fcfb9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" Jul 28 15:34:08.484268 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-tmp\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.484268 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-config\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.484484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.484484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvhd\" (UniqueName: \"kubernetes.io/projected/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-kube-api-access-6qvhd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.484484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-tmp-dir\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.484484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-config-volume\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.484484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcd2\" (UniqueName: \"kubernetes.io/projected/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-kube-api-access-jmcd2\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.484714 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mct\" (UniqueName: \"kubernetes.io/projected/66dab572-5fdd-4a86-ba23-8b43f447b27a-kube-api-access-m4mct\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.484846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-tmp-dir\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.484846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.484846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-tmp\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.484968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.484968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.485031 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-config\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.485031 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.484978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-serving-cert\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.485091 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.485091 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-config-volume\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.485091 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.485178 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-snapshots\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.485178 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7n7\" (UniqueName: \"kubernetes.io/projected/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-kube-api-access-gm7n7\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.485237 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjs85\" (UniqueName: \"kubernetes.io/projected/832386bd-d9d4-477c-a341-d7187c9a08e5-kube-api-access-kjs85\") pod \"network-check-source-7d88cb55d8-rtpzf\" (UID: \"832386bd-d9d4-477c-a341-d7187c9a08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" Jul 28 15:34:08.485237 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djcr\" (UniqueName: \"kubernetes.io/projected/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-kube-api-access-4djcr\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.485300 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66dab572-5fdd-4a86-ba23-8b43f447b27a-cert\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.485852 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.485464 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:08.485852 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.485523 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.985505805 +0000 UTC m=+35.666562801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:08.485852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.485852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.485616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.486201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.486181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-snapshots\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.486201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.486195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.487471 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.487448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.487471 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.487459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-serving-cert\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.487689 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.487670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66dab572-5fdd-4a86-ba23-8b43f447b27a-cert\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.487739 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.487720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.493902 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.493528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvhd\" (UniqueName: \"kubernetes.io/projected/398680a4-2b5a-4ee2-9a9f-3ec51e80043c-kube-api-access-6qvhd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9\" (UID: \"398680a4-2b5a-4ee2-9a9f-3ec51e80043c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.494166 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.494102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcd2\" (UniqueName: \"kubernetes.io/projected/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-kube-api-access-jmcd2\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.495575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.495552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mct\" (UniqueName: \"kubernetes.io/projected/66dab572-5fdd-4a86-ba23-8b43f447b27a-kube-api-access-m4mct\") pod \"ingress-canary-j5ftk\" (UID: \"66dab572-5fdd-4a86-ba23-8b43f447b27a\") " pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.496082 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.496059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7n7\" (UniqueName: \"kubernetes.io/projected/ad57ff0f-57a8-4509-9a41-329fa0b76aa4-kube-api-access-gm7n7\") pod \"insights-operator-5885fdb759-lgx6t\" (UID: \"ad57ff0f-57a8-4509-9a41-329fa0b76aa4\") " pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.496140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.496108 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjs85\" (UniqueName: \"kubernetes.io/projected/832386bd-d9d4-477c-a341-d7187c9a08e5-kube-api-access-kjs85\") pod \"network-check-source-7d88cb55d8-rtpzf\" (UID: \"832386bd-d9d4-477c-a341-d7187c9a08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" Jul 28 15:34:08.496193 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.496179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djcr\" (UniqueName: \"kubernetes.io/projected/c3909d0f-91ee-4bf5-ada5-bfc4f4362102-kube-api-access-4djcr\") pod \"service-ca-operator-5d6c75f84f-4rqn4\" (UID: \"c3909d0f-91ee-4bf5-ada5-bfc4f4362102\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.501028 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.501012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" Jul 28 15:34:08.580612 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.580536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:08.591843 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.591815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" Jul 28 15:34:08.598544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.598183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" Jul 28 15:34:08.602783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.602725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" Jul 28 15:34:08.614383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.614358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" Jul 28 15:34:08.618878 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.618854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j5ftk" Jul 28 15:34:08.717422 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.717311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b"] Jul 28 15:34:08.813854 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.813829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4"] Jul 28 15:34:08.816082 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:08.816049 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3909d0f_91ee_4bf5_ada5_bfc4f4362102.slice/crio-15620d607804789424a10d856fcffb3ea4b97cd0eee6a5e7256153bcc904741a WatchSource:0}: Error finding container 15620d607804789424a10d856fcffb3ea4b97cd0eee6a5e7256153bcc904741a: Status 404 returned error can't find the container with id 15620d607804789424a10d856fcffb3ea4b97cd0eee6a5e7256153bcc904741a Jul 28 15:34:08.833008 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.832718 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-s6mqj"] Jul 28 15:34:08.843258 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:08.843230 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891844af_0fb5_4513_b06b_4c23d070cc2e.slice/crio-33ea5616dacf0fcbd86d73b8c03c517fddd7c0c96779a2ddea492f41da815994 WatchSource:0}: Error finding container 33ea5616dacf0fcbd86d73b8c03c517fddd7c0c96779a2ddea492f41da815994: Status 404 returned error can't find the container with id 33ea5616dacf0fcbd86d73b8c03c517fddd7c0c96779a2ddea492f41da815994 Jul 28 15:34:08.853434 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.852898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:08.855635 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.855615 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j5ftk"] Jul 28 15:34:08.856516 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.856329 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmhl6\"" Jul 28 15:34:08.856516 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.856482 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 15:34:08.861571 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:08.861551 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dab572_5fdd_4a86_ba23_8b43f447b27a.slice/crio-74c825ce71dee762c8acd6f55691997c26582c88c9cc2bb5f45e9152059ee282 WatchSource:0}: Error finding container 74c825ce71dee762c8acd6f55691997c26582c88c9cc2bb5f45e9152059ee282: Status 404 returned error can't find the container with id 74c825ce71dee762c8acd6f55691997c26582c88c9cc2bb5f45e9152059ee282 Jul 28 15:34:08.881557 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.881540 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9"] Jul 28 15:34:08.883476 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:08.883458 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398680a4_2b5a_4ee2_9a9f_3ec51e80043c.slice/crio-5f1e7c1842a16558f74ce102f03be5c8ef62584086bec7a774d9dd18e8f414bb WatchSource:0}: Error finding container 5f1e7c1842a16558f74ce102f03be5c8ef62584086bec7a774d9dd18e8f414bb: Status 404 returned error can't find the container with id 5f1e7c1842a16558f74ce102f03be5c8ef62584086bec7a774d9dd18e8f414bb Jul 28 15:34:08.890944 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.890924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:08.891036 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.891018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.891100 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891067 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:08.891100 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.891085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:08.891192 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891128 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.891108689 +0000 UTC m=+36.572165698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:08.891192 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891130 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:08.891192 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891170 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.891159149 +0000 UTC m=+36.572216145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:08.891192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.891169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891200 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.891183978 +0000 UTC m=+36.572240973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891221 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.891222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891251 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.891241136 +0000 UTC m=+36.572298130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891287 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891301 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:08.891406 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.891351 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.89133663 +0000 UTC m=+36.572393621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:08.992161 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:08.992126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:08.992341 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.992285 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:08.992385 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:08.992350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:09.992329857 +0000 UTC m=+36.673386846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:09.060586 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.060554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" event={"ID":"2579dcd3-2e50-45eb-bef3-de71d89fcfb9","Type":"ContainerStarted","Data":"73dc780ff0b086b736a2a29e4dbd13c5f977a7a6e194d79d445d0ccba40a2dfc"} Jul 28 15:34:09.061543 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.061517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" event={"ID":"398680a4-2b5a-4ee2-9a9f-3ec51e80043c","Type":"ContainerStarted","Data":"5f1e7c1842a16558f74ce102f03be5c8ef62584086bec7a774d9dd18e8f414bb"} Jul 28 15:34:09.062643 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.062619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" event={"ID":"c3909d0f-91ee-4bf5-ada5-bfc4f4362102","Type":"ContainerStarted","Data":"15620d607804789424a10d856fcffb3ea4b97cd0eee6a5e7256153bcc904741a"} Jul 28 15:34:09.065722 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.065694 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ee68990-196b-47a9-bf0a-397cc21718da" containerID="49683385045d6380fb6f58cc3b3902a1a82e7df07d26cf05a26006719262b39c" exitCode=0 Jul 28 15:34:09.065827 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.065754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerDied","Data":"49683385045d6380fb6f58cc3b3902a1a82e7df07d26cf05a26006719262b39c"} Jul 28 15:34:09.066965 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.066941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j5ftk" event={"ID":"66dab572-5fdd-4a86-ba23-8b43f447b27a","Type":"ContainerStarted","Data":"74c825ce71dee762c8acd6f55691997c26582c88c9cc2bb5f45e9152059ee282"} Jul 28 15:34:09.067952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.067934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" event={"ID":"891844af-0fb5-4513-b06b-4c23d070cc2e","Type":"ContainerStarted","Data":"33ea5616dacf0fcbd86d73b8c03c517fddd7c0c96779a2ddea492f41da815994"} Jul 28 15:34:09.087773 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.087745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgx6t"] Jul 28 15:34:09.088490 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.088471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf"] Jul 28 15:34:09.089841 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:09.089819 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad57ff0f_57a8_4509_9a41_329fa0b76aa4.slice/crio-270c531757f3fc946386b34fd64a299cc37df50acb504b083ed3206add15fe28 WatchSource:0}: Error finding container 270c531757f3fc946386b34fd64a299cc37df50acb504b083ed3206add15fe28: Status 404 returned error can't find the container with id 270c531757f3fc946386b34fd64a299cc37df50acb504b083ed3206add15fe28 Jul 28 15:34:09.092940 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:09.092916 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832386bd_d9d4_477c_a341_d7187c9a08e5.slice/crio-a41e0d3c522502f797cae2301829aea1188ece368847c23f51f0ccb34cc7caaa WatchSource:0}: Error finding container a41e0d3c522502f797cae2301829aea1188ece368847c23f51f0ccb34cc7caaa: Status 404 returned error can't find the container with id a41e0d3c522502f797cae2301829aea1188ece368847c23f51f0ccb34cc7caaa Jul 28 15:34:09.852952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.852917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:09.853934 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.853566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:09.857075 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.856312 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 15:34:09.858876 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.858856 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptv7b\"" Jul 28 15:34:09.899191 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.899156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:09.899361 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.899237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:09.899361 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.899286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:09.899361 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.899313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:09.899528 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:09.899365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:09.899528 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.899522 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:09.899623 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.899586 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.899567438 +0000 UTC m=+38.580624436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:09.900050 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900029 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:09.900150 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900080 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.900064408 +0000 UTC m=+38.581121403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:09.900150 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900145 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.90013438 +0000 UTC m=+38.581191384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:09.900274 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900205 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:09.900274 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900237 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.900226758 +0000 UTC m=+38.581283764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:09.900378 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900299 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:09.900378 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900309 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:09.900378 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:09.900338 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.900327895 +0000 UTC m=+38.581384901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:10.000644 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:10.000606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:10.001376 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:10.000939 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:10.001376 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:10.001005 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:12.000986406 +0000 UTC m=+38.682043400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:10.086474 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:10.085931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" event={"ID":"4ee68990-196b-47a9-bf0a-397cc21718da","Type":"ContainerStarted","Data":"cd795ed35b7b981a5c49dc34de2f8b4b0413ef3516f87b4cedf6454e49a554cc"} Jul 28 15:34:10.088079 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:10.088008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" event={"ID":"ad57ff0f-57a8-4509-9a41-329fa0b76aa4","Type":"ContainerStarted","Data":"270c531757f3fc946386b34fd64a299cc37df50acb504b083ed3206add15fe28"} Jul 28 15:34:10.093230 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:10.093177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" event={"ID":"832386bd-d9d4-477c-a341-d7187c9a08e5","Type":"ContainerStarted","Data":"a41e0d3c522502f797cae2301829aea1188ece368847c23f51f0ccb34cc7caaa"} Jul 28 15:34:10.112554 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:10.112178 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vbs4j" podStartSLOduration=8.559061187 podStartE2EDuration="37.112159974s" totalImagesPullingTime="28.553098787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.109529605 +0000 UTC m=+36.790586618" watchObservedRunningTime="2026-07-28 15:34:10.112159974 +0000 UTC m=+36.793216987" Jul 28 15:34:11.101625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.101144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" event={"ID":"2579dcd3-2e50-45eb-bef3-de71d89fcfb9","Type":"ContainerStarted","Data":"0d534bf4abf9bdc0bceeab4e45cf44cc15f056819562853761c79ab3b5439278"} Jul 28 15:34:11.921377 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.921341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:11.921560 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.921388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:11.921560 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.921409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:11.921560 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.921454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:11.921560 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921544 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.921519618 +0000 UTC m=+42.602576614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921565 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:11.921578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921584 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921603 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921572 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921622 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.921614246 +0000 UTC m=+42.602671236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921645 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.921630662 +0000 UTC m=+42.602687653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921671 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921694 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.921682578 +0000 UTC m=+42.602739570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:11.921780 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:11.921712 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.921702316 +0000 UTC m=+42.602759349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:12.022992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:12.022788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:12.022992 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:12.022938 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:12.023232 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:12.023007 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:16.022988558 +0000 UTC m=+42.704045555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:15.956002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:15.955958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:15.956033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:15.956063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:15.956108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956141 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.956120774 +0000 UTC m=+50.637177788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956194 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956212 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:15.956227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956234 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.956220846 +0000 UTC m=+50.637277839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956238 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956259 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956283 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.956261996 +0000 UTC m=+50.637318997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956302 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.956307203 +0000 UTC m=+50.637364197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:15.956728 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:15.956349 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.956339524 +0000 UTC m=+50.637396517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:16.057406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:16.057363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:16.057555 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:16.057514 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:16.057594 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:16.057568 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:24.057554811 +0000 UTC m=+50.738611801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:17.126241 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.125718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" event={"ID":"ad57ff0f-57a8-4509-9a41-329fa0b76aa4","Type":"ContainerStarted","Data":"9ad8310994e855a5293aea28e6bf6ea6b1de227c29b7201fcb622165b4ac03a5"} Jul 28 15:34:17.127725 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.127659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" event={"ID":"398680a4-2b5a-4ee2-9a9f-3ec51e80043c","Type":"ContainerStarted","Data":"fc4c66e64af5e7a46e696d9c73662c74f7404c7be608f6f6173b1a9dcd3cf13b"} Jul 28 15:34:17.129236 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.129211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" event={"ID":"c3909d0f-91ee-4bf5-ada5-bfc4f4362102","Type":"ContainerStarted","Data":"c1874137fcef420dba093ec89877afdd99a99bcab85b507aac7342107f3bcc58"} Jul 28 15:34:17.130952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.130916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" event={"ID":"832386bd-d9d4-477c-a341-d7187c9a08e5","Type":"ContainerStarted","Data":"5427b1e046d297c1f1df5e923c9d07da8da81fd63af02d7073323e25d840bb19"} Jul 28 15:34:17.132880 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.132858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j5ftk" event={"ID":"66dab572-5fdd-4a86-ba23-8b43f447b27a","Type":"ContainerStarted","Data":"048e150a2eae9e0bc8d1d23d63c9aec7318c1ef5662ffd6f979b3ceaf2b75400"} Jul 28 15:34:17.134443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.134425 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/0.log" Jul 28 15:34:17.134538 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.134464 2580 generic.go:356] "Generic (PLEG): container finished" podID="891844af-0fb5-4513-b06b-4c23d070cc2e" containerID="6fab72b99984b4a6a9de77ec9f78df6e17a78199db351eb311569d9edd04550d" exitCode=255 Jul 28 15:34:17.134538 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.134498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" event={"ID":"891844af-0fb5-4513-b06b-4c23d070cc2e","Type":"ContainerDied","Data":"6fab72b99984b4a6a9de77ec9f78df6e17a78199db351eb311569d9edd04550d"} Jul 28 15:34:17.134717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.134701 2580 scope.go:117] "RemoveContainer" containerID="6fab72b99984b4a6a9de77ec9f78df6e17a78199db351eb311569d9edd04550d" Jul 28 15:34:17.144970 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.144930 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-h5h2b" podStartSLOduration=24.06916626 podStartE2EDuration="26.144917314s" totalImagesPullingTime="2.075751054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:11.120638681 +0000 UTC m=+37.801695693" watchObservedRunningTime="2026-07-28 15:34:17.144917314 +0000 UTC m=+43.825974328" Jul 28 15:34:17.145077 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.145003 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" podStartSLOduration=24.753948484 podStartE2EDuration="32.144999138s" totalImagesPullingTime="7.391050654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.144020419 +0000 UTC m=+43.825077434" watchObservedRunningTime="2026-07-28 15:34:17.144999138 +0000 UTC m=+43.826056128" Jul 28 15:34:17.182728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.182681 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-rtpzf" podStartSLOduration=11.78501728 podStartE2EDuration="19.182661516s" totalImagesPullingTime="7.397644236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.181667081 +0000 UTC m=+43.862724096" watchObservedRunningTime="2026-07-28 15:34:17.182661516 +0000 UTC m=+43.863718520" Jul 28 15:34:17.200324 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.200273 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" podStartSLOduration=14.538015773 podStartE2EDuration="22.200252823s" totalImagesPullingTime="7.66223705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.199428756 +0000 UTC m=+43.880485772" watchObservedRunningTime="2026-07-28 15:34:17.200252823 +0000 UTC m=+43.881309836" Jul 28 15:34:17.216382 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.216332 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" podStartSLOduration=14.619004316 podStartE2EDuration="22.216314476s" totalImagesPullingTime="7.59731016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.215737916 +0000 UTC m=+43.896794929" watchObservedRunningTime="2026-07-28 15:34:17.216314476 +0000 UTC m=+43.897371489" Jul 28 15:34:17.232129 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.232082 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j5ftk" podStartSLOduration=1.613397972 podStartE2EDuration="9.232065101s" totalImagesPullingTime="7.618667129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.231235505 +0000 UTC m=+43.912292520" watchObservedRunningTime="2026-07-28 15:34:17.232065101 +0000 UTC m=+43.913122118" Jul 28 15:34:17.865925 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.865329 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kdn8k"] Jul 28 15:34:17.896045 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.896011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kdn8k"] Jul 28 15:34:17.896190 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.896124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:17.899143 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.899123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgdjb\"" Jul 28 15:34:17.899281 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.899257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 15:34:17.900585 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:17.900554 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 15:34:18.077611 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.077579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.078031 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.077995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1889b55-d339-4731-a314-e58ea353cf0c-crio-socket\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.078146 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.078069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1889b55-d339-4731-a314-e58ea353cf0c-data-volume\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.078146 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.078131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ppf\" (UniqueName: \"kubernetes.io/projected/c1889b55-d339-4731-a314-e58ea353cf0c-kube-api-access-l7ppf\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.078271 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.078212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.140777 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.140706 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:34:18.141209 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.141195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/0.log" Jul 28 15:34:18.141266 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.141227 2580 generic.go:356] "Generic (PLEG): container finished" podID="891844af-0fb5-4513-b06b-4c23d070cc2e" containerID="42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320" exitCode=255 Jul 28 15:34:18.141355 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.141327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" event={"ID":"891844af-0fb5-4513-b06b-4c23d070cc2e","Type":"ContainerDied","Data":"42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320"} Jul 28 15:34:18.141456 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.141387 2580 scope.go:117] "RemoveContainer" containerID="6fab72b99984b4a6a9de77ec9f78df6e17a78199db351eb311569d9edd04550d" Jul 28 15:34:18.141615 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.141599 2580 scope.go:117] "RemoveContainer" containerID="42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320" Jul 28 15:34:18.141822 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.141771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-s6mqj_openshift-console-operator(891844af-0fb5-4513-b06b-4c23d070cc2e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" podUID="891844af-0fb5-4513-b06b-4c23d070cc2e" Jul 28 15:34:18.178979 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.178943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ppf\" (UniqueName: \"kubernetes.io/projected/c1889b55-d339-4731-a314-e58ea353cf0c-kube-api-access-l7ppf\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1889b55-d339-4731-a314-e58ea353cf0c-crio-socket\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179155 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.179150 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:34:18.179364 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1889b55-d339-4731-a314-e58ea353cf0c-data-volume\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179364 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.179214 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.679195155 +0000 UTC m=+45.360252145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : secret "insights-runtime-extractor-tls" not found Jul 28 15:34:18.179364 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.179213 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:34:18.179364 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.179291 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.679275097 +0000 UTC m=+45.360332103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : configmap "kube-rbac-proxy" not found Jul 28 15:34:18.179564 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1889b55-d339-4731-a314-e58ea353cf0c-crio-socket\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.179564 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.179509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1889b55-d339-4731-a314-e58ea353cf0c-data-volume\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.189240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.189217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ppf\" (UniqueName: \"kubernetes.io/projected/c1889b55-d339-4731-a314-e58ea353cf0c-kube-api-access-l7ppf\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.580696 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.580659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:18.580880 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.580760 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:18.684839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.684789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:18.685026 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.684926 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:34:18.685026 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.684995 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.684974469 +0000 UTC m=+46.366031459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : configmap "kube-rbac-proxy" not found Jul 28 15:34:18.685026 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.684999 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:34:18.685154 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:18.685044 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.685033757 +0000 UTC m=+46.366090747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : secret "insights-runtime-extractor-tls" not found Jul 28 15:34:18.685154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:18.684931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:19.146596 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:19.146567 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:34:19.147003 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:19.146922 2580 scope.go:117] "RemoveContainer" containerID="42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320" Jul 28 15:34:19.147096 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:19.147077 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-s6mqj_openshift-console-operator(891844af-0fb5-4513-b06b-4c23d070cc2e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" podUID="891844af-0fb5-4513-b06b-4c23d070cc2e" Jul 28 15:34:19.694643 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:19.694600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:19.694841 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:19.694656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:19.694841 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:19.694717 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:34:19.694841 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:19.694759 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:34:19.694952 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:19.694857 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.694837781 +0000 UTC m=+48.375894778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : secret "insights-runtime-extractor-tls" not found Jul 28 15:34:19.694952 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:19.694882 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.694875431 +0000 UTC m=+48.375932420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : configmap "kube-rbac-proxy" not found Jul 28 15:34:19.853584 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:19.853550 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xz94w_4e57260c-da09-4286-8a99-5739e4d5d6d5/dns-node-resolver/0.log" Jul 28 15:34:20.149378 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:20.149307 2580 scope.go:117] "RemoveContainer" containerID="42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320" Jul 28 15:34:20.149730 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:20.149460 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-s6mqj_openshift-console-operator(891844af-0fb5-4513-b06b-4c23d070cc2e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" podUID="891844af-0fb5-4513-b06b-4c23d070cc2e" Jul 28 15:34:20.853097 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:20.853069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85bv6_5968e5c7-27a6-4b49-b0af-d89ee40cdeab/node-ca/0.log" Jul 28 15:34:21.711142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:21.711108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:21.711525 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:21.711230 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 15:34:21.711525 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:21.711284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:21.711525 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:21.711337 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:25.711287218 +0000 UTC m=+52.392344227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : configmap "kube-rbac-proxy" not found Jul 28 15:34:21.711525 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:21.711383 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:34:21.711525 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:21.711430 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:25.711417763 +0000 UTC m=+52.392474754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : secret "insights-runtime-extractor-tls" not found Jul 28 15:34:21.853075 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:21.853045 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j5ftk_66dab572-5fdd-4a86-ba23-8b43f447b27a/serve-healthcheck-canary/0.log" Jul 28 15:34:24.037333 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.037297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.037365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.037391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.037409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.037435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037496 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037539 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037543 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037564 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bff99df44-qlw9d: secret "image-registry-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037582 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.037560956 +0000 UTC m=+66.718617947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : secret "router-metrics-certs-default" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037541 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037604 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls podName:61375885-2520-41d7-a030-394abf63bba1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.037593863 +0000 UTC m=+66.718650854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls") pod "cluster-samples-operator-6748489656-w4rqw" (UID: "61375885-2520-41d7-a030-394abf63bba1") : secret "samples-operator-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037628 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls podName:e8143238-a595-4467-81b2-0ee06280cd4d nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.037610111 +0000 UTC m=+66.718667101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lwjpp" (UID: "e8143238-a595-4467-81b2-0ee06280cd4d") : secret "cluster-monitoring-operator-tls" not found Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037644 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle podName:8975da34-3728-4e3d-8f6a-e782583c3e94 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.03763711 +0000 UTC m=+66.718694099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle") pod "router-default-57cdcfc679-kw468" (UID: "8975da34-3728-4e3d-8f6a-e782583c3e94") : configmap references non-existent config key: service-ca.crt Jul 28 15:34:24.037851 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.037660 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls podName:99d0f269-fecd-4743-9e85-1618f741ccae nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.037653995 +0000 UTC m=+66.718710984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls") pod "image-registry-bff99df44-qlw9d" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae") : secret "image-registry-tls" not found Jul 28 15:34:24.137774 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:24.137740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:24.137965 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.137897 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 15:34:24.137965 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:24.137963 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls podName:92fdb4d9-0e11-4c05-99fc-39cb652c4d98 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:40.137944593 +0000 UTC m=+66.819001586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls") pod "dns-default-5mtj5" (UID: "92fdb4d9-0e11-4c05-99fc-39cb652c4d98") : secret "dns-default-metrics-tls" not found Jul 28 15:34:25.749175 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:25.749141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:25.749596 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:25.749193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:25.749596 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:25.749292 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 15:34:25.749596 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:34:25.749353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls podName:c1889b55-d339-4731-a314-e58ea353cf0c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:33.749338376 +0000 UTC m=+60.430395366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kdn8k" (UID: "c1889b55-d339-4731-a314-e58ea353cf0c") : secret "insights-runtime-extractor-tls" not found Jul 28 15:34:25.749732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:25.749716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1889b55-d339-4731-a314-e58ea353cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:32.853625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:32.853589 2580 scope.go:117] "RemoveContainer" containerID="42ecf1778fb87537fe9b551add46cb5bb8e4a3246f3fa617f543b945252a7320" Jul 28 15:34:33.191476 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.191449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:34:33.191636 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.191531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" event={"ID":"891844af-0fb5-4513-b06b-4c23d070cc2e","Type":"ContainerStarted","Data":"fde88cf6ce4d526dcfc1c08468c98a5bf8d5aa59446e510b1d63a7a5867de916"} Jul 28 15:34:33.191854 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.191832 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:33.213190 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.213132 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" podStartSLOduration=32.57594451 podStartE2EDuration="40.213113074s" totalImagesPullingTime="7.637168564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:33.211721087 +0000 UTC m=+59.892778097" watchObservedRunningTime="2026-07-28 15:34:33.213113074 +0000 UTC m=+59.894170084" Jul 28 15:34:33.817908 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.817881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:33.820402 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.820376 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1889b55-d339-4731-a314-e58ea353cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kdn8k\" (UID: \"c1889b55-d339-4731-a314-e58ea353cf0c\") " pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:33.839315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:33.839291 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-s6mqj" Jul 28 15:34:34.054851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:34.054826 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hv4d" Jul 28 15:34:34.109433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:34.109359 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgdjb\"" Jul 28 15:34:34.117050 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:34.117034 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kdn8k" Jul 28 15:34:34.247596 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:34.247563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kdn8k"] Jul 28 15:34:34.251716 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:34.251688 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1889b55_d339_4731_a314_e58ea353cf0c.slice/crio-a6f2976387c49b9599165a7a351e8950d35ae52d70ee7e7116b83179e42fd59a WatchSource:0}: Error finding container a6f2976387c49b9599165a7a351e8950d35ae52d70ee7e7116b83179e42fd59a: Status 404 returned error can't find the container with id a6f2976387c49b9599165a7a351e8950d35ae52d70ee7e7116b83179e42fd59a Jul 28 15:34:35.203462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:35.203425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdn8k" event={"ID":"c1889b55-d339-4731-a314-e58ea353cf0c","Type":"ContainerStarted","Data":"a6f2976387c49b9599165a7a351e8950d35ae52d70ee7e7116b83179e42fd59a"} Jul 28 15:34:39.675747 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.675710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:39.676132 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.675758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:39.678968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.678944 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 15:34:39.679096 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.678985 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 15:34:39.688624 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.688596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29e3fd0b-5634-4368-a41a-2fa9768aa695-original-pull-secret\") pod \"global-pull-secret-syncer-cbcgv\" (UID: \"29e3fd0b-5634-4368-a41a-2fa9768aa695\") " pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:39.688624 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.688619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c18d1a-bf4d-4ced-987a-78745c891505-metrics-certs\") pod \"network-metrics-daemon-gs4v8\" (UID: \"79c18d1a-bf4d-4ced-987a-78745c891505\") " pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:39.769267 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.769245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmhl6\"" Jul 28 15:34:39.775907 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.775884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gs4v8" Jul 28 15:34:39.776192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.776175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:39.778890 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.778789 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtq2\" (UniqueName: \"kubernetes.io/projected/727244f6-be9a-4e62-8294-29ef4c4d7936-kube-api-access-2xtq2\") pod \"network-check-target-hxplb\" (UID: \"727244f6-be9a-4e62-8294-29ef4c4d7936\") " pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:39.876319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.876289 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptv7b\"" Jul 28 15:34:39.880575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.880550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:39.881911 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.881848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbcgv" Jul 28 15:34:39.924665 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:39.924542 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c18d1a_bf4d_4ced_987a_78745c891505.slice/crio-17b1afbcd8e766408e5e121a4a90d78f7cee9b13cfedd198394bf7c6c4bab770 WatchSource:0}: Error finding container 17b1afbcd8e766408e5e121a4a90d78f7cee9b13cfedd198394bf7c6c4bab770: Status 404 returned error can't find the container with id 17b1afbcd8e766408e5e121a4a90d78f7cee9b13cfedd198394bf7c6c4bab770 Jul 28 15:34:39.924665 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:39.924607 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gs4v8"] Jul 28 15:34:40.018261 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.018226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbcgv"] Jul 28 15:34:40.021081 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:40.021032 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e3fd0b_5634_4368_a41a_2fa9768aa695.slice/crio-3d7d680fe601c5343ef65d64098ca9d6245d0b2a4d85d5642daa82c844a72bd7 WatchSource:0}: Error finding container 3d7d680fe601c5343ef65d64098ca9d6245d0b2a4d85d5642daa82c844a72bd7: Status 404 returned error can't find the container with id 3d7d680fe601c5343ef65d64098ca9d6245d0b2a4d85d5642daa82c844a72bd7 Jul 28 15:34:40.034192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.034165 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxplb"] Jul 28 15:34:40.037232 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:40.037205 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727244f6_be9a_4e62_8294_29ef4c4d7936.slice/crio-a5d54acaba91a9edce33a7031bfd08bf7b8c24e586d55d293c0ff145861c423a WatchSource:0}: Error finding container a5d54acaba91a9edce33a7031bfd08bf7b8c24e586d55d293c0ff145861c423a: Status 404 returned error can't find the container with id a5d54acaba91a9edce33a7031bfd08bf7b8c24e586d55d293c0ff145861c423a Jul 28 15:34:40.077828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.077785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:40.077984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.077856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:40.077984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.077886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:40.077984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.077911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:40.077984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.077941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:40.078556 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.078525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8975da34-3728-4e3d-8f6a-e782583c3e94-service-ca-bundle\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:40.080369 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.080343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8975da34-3728-4e3d-8f6a-e782583c3e94-metrics-certs\") pod \"router-default-57cdcfc679-kw468\" (UID: \"8975da34-3728-4e3d-8f6a-e782583c3e94\") " pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:40.080469 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.080441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/61375885-2520-41d7-a030-394abf63bba1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-w4rqw\" (UID: \"61375885-2520-41d7-a030-394abf63bba1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:40.081005 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.080985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"image-registry-bff99df44-qlw9d\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:40.081059 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.081002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8143238-a595-4467-81b2-0ee06280cd4d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lwjpp\" (UID: \"e8143238-a595-4467-81b2-0ee06280cd4d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:40.089469 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.089449 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5rqzv\"" Jul 28 15:34:40.096937 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.096916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" Jul 28 15:34:40.179409 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.179328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:40.181694 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.181674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92fdb4d9-0e11-4c05-99fc-39cb652c4d98-metrics-tls\") pod \"dns-default-5mtj5\" (UID: \"92fdb4d9-0e11-4c05-99fc-39cb652c4d98\") " pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:40.224550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.224514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbcgv" event={"ID":"29e3fd0b-5634-4368-a41a-2fa9768aa695","Type":"ContainerStarted","Data":"3d7d680fe601c5343ef65d64098ca9d6245d0b2a4d85d5642daa82c844a72bd7"} Jul 28 15:34:40.225000 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.224970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp"] Jul 28 15:34:40.226270 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.226211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxplb" event={"ID":"727244f6-be9a-4e62-8294-29ef4c4d7936","Type":"ContainerStarted","Data":"caa6871a24ab5b34d056aaf0a37aa0e3b3ac47672ce6fafc2a52e2fa5a7e4adb"} Jul 28 15:34:40.226270 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.226255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxplb" event={"ID":"727244f6-be9a-4e62-8294-29ef4c4d7936","Type":"ContainerStarted","Data":"a5d54acaba91a9edce33a7031bfd08bf7b8c24e586d55d293c0ff145861c423a"} Jul 28 15:34:40.226386 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.226365 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:34:40.227833 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.227815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdn8k" event={"ID":"c1889b55-d339-4731-a314-e58ea353cf0c","Type":"ContainerStarted","Data":"ce1727beb279959b3f772308045707fe8c7756169bdaa18a7cc540716bcceec2"} Jul 28 15:34:40.229974 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:40.229930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8143238_a595_4467_81b2_0ee06280cd4d.slice/crio-faa0f9348ab534ec013d91e2894642cda171269d96f74bb58af96b85f8323c8c WatchSource:0}: Error finding container faa0f9348ab534ec013d91e2894642cda171269d96f74bb58af96b85f8323c8c: Status 404 returned error can't find the container with id faa0f9348ab534ec013d91e2894642cda171269d96f74bb58af96b85f8323c8c Jul 28 15:34:40.231668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.231646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs4v8" event={"ID":"79c18d1a-bf4d-4ced-987a-78745c891505","Type":"ContainerStarted","Data":"17b1afbcd8e766408e5e121a4a90d78f7cee9b13cfedd198394bf7c6c4bab770"} Jul 28 15:34:40.242948 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.242895 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hxplb" podStartSLOduration=66.242876872 podStartE2EDuration="1m6.242876872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:40.242058786 +0000 UTC m=+66.923115812" watchObservedRunningTime="2026-07-28 15:34:40.242876872 +0000 UTC m=+66.923933862" Jul 28 15:34:40.282952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.282920 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxdc4\"" Jul 28 15:34:40.291276 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.291254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:40.296653 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.296635 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jpm9g\"" Jul 28 15:34:40.304122 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.304100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:40.310659 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.310641 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l6dkt\"" Jul 28 15:34:40.318188 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.318135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" Jul 28 15:34:40.412409 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.411959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sv4z4\"" Jul 28 15:34:40.418783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.418756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:40.463667 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.463630 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:34:40.470878 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:40.470841 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d0f269_fecd_4743_9e85_1618f741ccae.slice/crio-3c055abf1c99375d2435d4c6623c2a045e88842fd86e630bee8abe1a79d2e1be WatchSource:0}: Error finding container 3c055abf1c99375d2435d4c6623c2a045e88842fd86e630bee8abe1a79d2e1be: Status 404 returned error can't find the container with id 3c055abf1c99375d2435d4c6623c2a045e88842fd86e630bee8abe1a79d2e1be Jul 28 15:34:40.502627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.502413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57cdcfc679-kw468"] Jul 28 15:34:40.531374 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.531260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw"] Jul 28 15:34:40.612502 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:40.612427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5mtj5"] Jul 28 15:34:40.617540 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:40.617065 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fdb4d9_0e11_4c05_99fc_39cb652c4d98.slice/crio-f697862f309e522cb86b21887574c4b270a89088a290d49b302edac396ed349e WatchSource:0}: Error finding container f697862f309e522cb86b21887574c4b270a89088a290d49b302edac396ed349e: Status 404 returned error can't find the container with id f697862f309e522cb86b21887574c4b270a89088a290d49b302edac396ed349e Jul 28 15:34:41.246761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.245971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57cdcfc679-kw468" event={"ID":"8975da34-3728-4e3d-8f6a-e782583c3e94","Type":"ContainerStarted","Data":"372c8439ad41e9974a0a7708c5cab9f4b3abf048691b09b7c1479cfae2d8cf63"} Jul 28 15:34:41.246761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.246032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57cdcfc679-kw468" event={"ID":"8975da34-3728-4e3d-8f6a-e782583c3e94","Type":"ContainerStarted","Data":"a05af8df9dc880717696617d0d76efe971788def9a0f9e4b9a881ac6be0a7a6f"} Jul 28 15:34:41.250714 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.250610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5mtj5" event={"ID":"92fdb4d9-0e11-4c05-99fc-39cb652c4d98","Type":"ContainerStarted","Data":"f697862f309e522cb86b21887574c4b270a89088a290d49b302edac396ed349e"} Jul 28 15:34:41.253825 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.253722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" event={"ID":"61375885-2520-41d7-a030-394abf63bba1","Type":"ContainerStarted","Data":"6b064417a8df71bcbabf2a9150240d3e17dac83f428268af247d0046b785341d"} Jul 28 15:34:41.259433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.258761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" event={"ID":"99d0f269-fecd-4743-9e85-1618f741ccae","Type":"ContainerStarted","Data":"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1"} Jul 28 15:34:41.259433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.258827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" event={"ID":"99d0f269-fecd-4743-9e85-1618f741ccae","Type":"ContainerStarted","Data":"3c055abf1c99375d2435d4c6623c2a045e88842fd86e630bee8abe1a79d2e1be"} Jul 28 15:34:41.259433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.259385 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:34:41.262241 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.262134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" event={"ID":"e8143238-a595-4467-81b2-0ee06280cd4d","Type":"ContainerStarted","Data":"faa0f9348ab534ec013d91e2894642cda171269d96f74bb58af96b85f8323c8c"} Jul 28 15:34:41.270375 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.269948 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-57cdcfc679-kw468" podStartSLOduration=56.269928449 podStartE2EDuration="56.269928449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:41.268747057 +0000 UTC m=+67.949804068" watchObservedRunningTime="2026-07-28 15:34:41.269928449 +0000 UTC m=+67.950985462" Jul 28 15:34:41.298105 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.297705 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" podStartSLOduration=67.297685642 podStartE2EDuration="1m7.297685642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:41.297121182 +0000 UTC m=+67.978178198" watchObservedRunningTime="2026-07-28 15:34:41.297685642 +0000 UTC m=+67.978742655" Jul 28 15:34:41.305289 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.305259 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:41.307960 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:41.307933 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:42.270856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:42.270657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs4v8" event={"ID":"79c18d1a-bf4d-4ced-987a-78745c891505","Type":"ContainerStarted","Data":"59fbf6a430d5ebf829fa7ed8a94639b9b9b4841e2b25fd2a4952540829620901"} Jul 28 15:34:42.270856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:42.270734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gs4v8" event={"ID":"79c18d1a-bf4d-4ced-987a-78745c891505","Type":"ContainerStarted","Data":"238cd8c00d05711e9a258f54a6a503123c7ce3928c690fa0ba05afc0969bc7d8"} Jul 28 15:34:42.271494 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:42.271471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:42.272902 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:42.272874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57cdcfc679-kw468" Jul 28 15:34:42.289589 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:42.289484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gs4v8" podStartSLOduration=66.462878524 podStartE2EDuration="1m8.289464075s" totalImagesPullingTime="1.826585551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:42.288168976 +0000 UTC m=+68.969226012" watchObservedRunningTime="2026-07-28 15:34:42.289464075 +0000 UTC m=+68.970521091" Jul 28 15:34:48.296077 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.296042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbcgv" event={"ID":"29e3fd0b-5634-4368-a41a-2fa9768aa695","Type":"ContainerStarted","Data":"0430bb17bbc67353ef86c2ab77938d11579510b922cc83c31ee0558b135ba094"} Jul 28 15:34:48.298033 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.298006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5mtj5" event={"ID":"92fdb4d9-0e11-4c05-99fc-39cb652c4d98","Type":"ContainerStarted","Data":"cf5ee756fe2538846505baa6f01a30f1a1490ecee502bbca18380fa5fd96028d"} Jul 28 15:34:48.299886 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.299864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" event={"ID":"61375885-2520-41d7-a030-394abf63bba1","Type":"ContainerStarted","Data":"2af2308f30775db1cd81b9efa7323236e5c3b6f195e33c3ba8155d9b660ae24d"} Jul 28 15:34:48.302465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.302384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" event={"ID":"e8143238-a595-4467-81b2-0ee06280cd4d","Type":"ContainerStarted","Data":"7ceb7faf7392c4895995995ee9e9945f46ad54b856414183174f72d51bbf3bff"} Jul 28 15:34:48.331828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.331756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cbcgv" podStartSLOduration=66.304675076 podStartE2EDuration="1m14.331738533s" totalImagesPullingTime="8.027063457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:48.313193692 +0000 UTC m=+74.994250706" watchObservedRunningTime="2026-07-28 15:34:48.331738533 +0000 UTC m=+75.012795547" Jul 28 15:34:48.332002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:48.331921 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lwjpp" podStartSLOduration=55.698462643 podStartE2EDuration="1m3.33191431s" totalImagesPullingTime="7.633451667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:48.331027147 +0000 UTC m=+75.012084160" watchObservedRunningTime="2026-07-28 15:34:48.33191431 +0000 UTC m=+75.012971325" Jul 28 15:34:49.308536 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.308501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdn8k" event={"ID":"c1889b55-d339-4731-a314-e58ea353cf0c","Type":"ContainerStarted","Data":"7ca790272bb7cef3024012f46589b3015d024a8ec13ac92573a7071670aaec1a"} Jul 28 15:34:49.310400 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.310373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5mtj5" event={"ID":"92fdb4d9-0e11-4c05-99fc-39cb652c4d98","Type":"ContainerStarted","Data":"5644a28a988a27d4517713de5416784aa574c0b1a58021ff1d429505e0b89c3d"} Jul 28 15:34:49.310544 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.310484 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5mtj5" Jul 28 15:34:49.316387 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.316355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" event={"ID":"61375885-2520-41d7-a030-394abf63bba1","Type":"ContainerStarted","Data":"60fda5d97c96373eb1d1c03667e5e1758e03aa9e6be626d2a23cc7f470f274ae"} Jul 28 15:34:49.347812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.347751 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5mtj5" podStartSLOduration=33.918628586 podStartE2EDuration="41.347730591s" totalImagesPullingTime="7.429102005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:49.347125617 +0000 UTC m=+76.028182629" watchObservedRunningTime="2026-07-28 15:34:49.347730591 +0000 UTC m=+76.028787605" Jul 28 15:34:49.381647 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:49.381590 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-w4rqw" podStartSLOduration=50.950168543 podStartE2EDuration="58.381573405s" totalImagesPullingTime="7.431404862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:49.380844145 +0000 UTC m=+76.061901157" watchObservedRunningTime="2026-07-28 15:34:49.381573405 +0000 UTC m=+76.062630417" Jul 28 15:34:51.936339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.936302 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:34:51.945303 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.945280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:51.948716 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.948610 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 15:34:51.948937 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.948782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 15:34:51.951075 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 15:34:51.951192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951081 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 15:34:51.951259 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951215 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 15:34:51.951259 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 15:34:51.951259 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 15:34:51.951791 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.951770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4ssqh\"" Jul 28 15:34:51.953208 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.953147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:34:51.956773 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:51.956751 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 15:34:52.086276 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlsb\" (UniqueName: \"kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.086460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.086446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187466 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187466 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187602 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187602 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187602 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187602 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.187810 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.187600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlsb\" (UniqueName: \"kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.188346 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.188292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.188505 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.188450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.188505 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.188487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.188823 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.188722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.190709 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.190679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.190709 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.190698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.197498 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.197473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlsb\" (UniqueName: \"kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb\") pod \"console-dc5db556d-djsgh\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.257194 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.257160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:34:52.329265 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.329221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kdn8k" event={"ID":"c1889b55-d339-4731-a314-e58ea353cf0c","Type":"ContainerStarted","Data":"91776bf964576cb50cc829cae96aa418b887c36fd2aa9c8071fd4e05e307f2f8"} Jul 28 15:34:52.350569 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.350518 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kdn8k" podStartSLOduration=17.651451635 podStartE2EDuration="35.350500825s" totalImagesPullingTime="17.69904919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:52.350100674 +0000 UTC m=+79.031157701" watchObservedRunningTime="2026-07-28 15:34:52.350500825 +0000 UTC m=+79.031557837" Jul 28 15:34:52.387911 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:52.387829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:34:52.390429 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:34:52.390401 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edcd7bb_b8fd_4ec8_ada4_da7e370a112a.slice/crio-8ec17af0bb429c9a7b2a6ef85a27a2f1031a91ec46f215b5e2545d81177b91f8 WatchSource:0}: Error finding container 8ec17af0bb429c9a7b2a6ef85a27a2f1031a91ec46f215b5e2545d81177b91f8: Status 404 returned error can't find the container with id 8ec17af0bb429c9a7b2a6ef85a27a2f1031a91ec46f215b5e2545d81177b91f8 Jul 28 15:34:53.335443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:53.335342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc5db556d-djsgh" event={"ID":"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a","Type":"ContainerStarted","Data":"8ec17af0bb429c9a7b2a6ef85a27a2f1031a91ec46f215b5e2545d81177b91f8"} Jul 28 15:34:55.345188 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:55.345153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc5db556d-djsgh" event={"ID":"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a","Type":"ContainerStarted","Data":"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b"} Jul 28 15:34:55.369007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:55.368945 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc5db556d-djsgh" podStartSLOduration=1.5680960879999999 podStartE2EDuration="4.368926572s" totalImagesPullingTime="2.800830484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:55.367837531 +0000 UTC m=+82.048894545" watchObservedRunningTime="2026-07-28 15:34:55.368926572 +0000 UTC m=+82.049983584" Jul 28 15:34:59.322347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:34:59.322314 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5mtj5" Jul 28 15:35:02.257865 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:02.257830 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:35:02.258239 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:02.257891 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:35:02.262625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:02.262608 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:35:02.376554 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:02.376529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:35:03.279261 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:03.279232 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:35:09.686505 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:09.686473 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:35:11.072928 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.072895 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t8b2k"] Jul 28 15:35:11.081899 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.081872 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg"] Jul 28 15:35:11.082038 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.081989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.085164 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.085146 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 15:35:11.085323 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.085170 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 15:35:11.085323 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.085289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.086510 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.086488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 15:35:11.086630 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.086537 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 15:35:11.086630 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.086499 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxmd6\"" Jul 28 15:35:11.087792 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.087776 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 15:35:11.088263 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.088245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 15:35:11.088351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.088294 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 15:35:11.088398 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.088377 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v94n9\"" Jul 28 15:35:11.093549 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.093527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg"] Jul 28 15:35:11.239483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-textfile\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.239483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-metrics-client-ca\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-root\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w252k\" (UniqueName: \"kubernetes.io/projected/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-api-access-w252k\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-wtmp\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.239732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-sys\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64fee8a1-9410-4b99-80e1-acf4cabbafc5-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s8v\" (UniqueName: \"kubernetes.io/projected/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-kube-api-access-h7s8v\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.240007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.239911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-tls\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.264793 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.264763 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hxplb" Jul 28 15:35:11.341130 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-wtmp\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341130 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341130 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-sys\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-wtmp\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64fee8a1-9410-4b99-80e1-acf4cabbafc5-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341509 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s8v\" (UniqueName: \"kubernetes.io/projected/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-kube-api-access-h7s8v\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341509 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-sys\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341509 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341509 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-tls\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-textfile\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-metrics-client-ca\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-root\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w252k\" (UniqueName: \"kubernetes.io/projected/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-api-access-w252k\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.341706 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64fee8a1-9410-4b99-80e1-acf4cabbafc5-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.342151 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.341835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.342221 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.342202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-textfile\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.342277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.342204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.342277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.342238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-metrics-client-ca\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.342397 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.342381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-root\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.343846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.343818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.344158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.344144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.344668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.344648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-node-exporter-tls\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.344825 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.344790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.351026 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.351007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s8v\" (UniqueName: \"kubernetes.io/projected/a31d5bef-2243-4d1e-8f68-a39f99cbe67f-kube-api-access-h7s8v\") pod \"node-exporter-t8b2k\" (UID: \"a31d5bef-2243-4d1e-8f68-a39f99cbe67f\") " pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.351359 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.351336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w252k\" (UniqueName: \"kubernetes.io/projected/64fee8a1-9410-4b99-80e1-acf4cabbafc5-kube-api-access-w252k\") pod \"kube-state-metrics-56d9f944b4-f89mg\" (UID: \"64fee8a1-9410-4b99-80e1-acf4cabbafc5\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.393390 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.393362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8b2k" Jul 28 15:35:11.399564 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.399540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" Jul 28 15:35:11.409032 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:35:11.409006 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31d5bef_2243_4d1e_8f68_a39f99cbe67f.slice/crio-e74e31234e7dd89c33fe51f661474123f2072e3db496f075f78d623e72814718 WatchSource:0}: Error finding container e74e31234e7dd89c33fe51f661474123f2072e3db496f075f78d623e72814718: Status 404 returned error can't find the container with id e74e31234e7dd89c33fe51f661474123f2072e3db496f075f78d623e72814718 Jul 28 15:35:11.536401 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:11.536371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg"] Jul 28 15:35:11.539359 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:35:11.539337 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fee8a1_9410_4b99_80e1_acf4cabbafc5.slice/crio-cf787a8910792471632fe6eea160b48203815d6c963d85c8499289adcd5f5026 WatchSource:0}: Error finding container cf787a8910792471632fe6eea160b48203815d6c963d85c8499289adcd5f5026: Status 404 returned error can't find the container with id cf787a8910792471632fe6eea160b48203815d6c963d85c8499289adcd5f5026 Jul 28 15:35:12.160389 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.160354 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 15:35:12.164421 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.164399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.168591 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.168537 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 15:35:12.168591 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.168583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 15:35:12.168769 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.168628 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 15:35:12.168942 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.168917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 15:35:12.169008 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.168947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 15:35:12.169072 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.169058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 15:35:12.169758 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.169738 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c94tx\"" Jul 28 15:35:12.169907 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.169780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 15:35:12.170044 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.169738 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 15:35:12.170165 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.170147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 15:35:12.181201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.181179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 15:35:12.352063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.351990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352235 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxbg\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-kube-api-access-hjxbg\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.352464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.352433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.412213 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.412179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8b2k" event={"ID":"a31d5bef-2243-4d1e-8f68-a39f99cbe67f","Type":"ContainerStarted","Data":"e74e31234e7dd89c33fe51f661474123f2072e3db496f075f78d623e72814718"} Jul 28 15:35:12.414813 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.414734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" event={"ID":"64fee8a1-9410-4b99-80e1-acf4cabbafc5","Type":"ContainerStarted","Data":"cf787a8910792471632fe6eea160b48203815d6c963d85c8499289adcd5f5026"} Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxbg\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-kube-api-access-hjxbg\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454020 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.453620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454738 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.454133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.454738 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:35:12.454239 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 15:35:12.454738 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:35:12.454300 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls podName:0ae083da-da86-4cf5-a7ac-fec1dcf6704d nodeName:}" failed. No retries permitted until 2026-07-28 15:35:12.954280177 +0000 UTC m=+99.635337169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0ae083da-da86-4cf5-a7ac-fec1dcf6704d") : secret "alertmanager-main-tls" not found Jul 28 15:35:12.454738 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.454687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.455333 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.455309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.456919 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.456893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.457837 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.457773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.461243 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.461221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.461589 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.461571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.462144 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.462093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.463442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.463417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.464036 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.463991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.464112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.464061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.466210 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.466169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxbg\" (UniqueName: \"kubernetes.io/projected/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-kube-api-access-hjxbg\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.957984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.957936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:12.960727 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:12.960705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ae083da-da86-4cf5-a7ac-fec1dcf6704d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ae083da-da86-4cf5-a7ac-fec1dcf6704d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:13.077099 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.077080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 15:35:13.265949 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.265787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 15:35:13.270261 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:35:13.270229 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae083da_da86_4cf5_a7ac_fec1dcf6704d.slice/crio-e53803bfd6408b4ac2cbe0e86af2c695563a80dd1703e5eb7e5a70e6f9b394ec WatchSource:0}: Error finding container e53803bfd6408b4ac2cbe0e86af2c695563a80dd1703e5eb7e5a70e6f9b394ec: Status 404 returned error can't find the container with id e53803bfd6408b4ac2cbe0e86af2c695563a80dd1703e5eb7e5a70e6f9b394ec Jul 28 15:35:13.421506 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.421482 2580 generic.go:356] "Generic (PLEG): container finished" podID="a31d5bef-2243-4d1e-8f68-a39f99cbe67f" containerID="757888c95e460fa65c41498a68fe7642ab85a55f4f2fbd1a206b24f75e66bf24" exitCode=0 Jul 28 15:35:13.421633 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.421571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8b2k" event={"ID":"a31d5bef-2243-4d1e-8f68-a39f99cbe67f","Type":"ContainerDied","Data":"757888c95e460fa65c41498a68fe7642ab85a55f4f2fbd1a206b24f75e66bf24"} Jul 28 15:35:13.423505 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.423477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" event={"ID":"64fee8a1-9410-4b99-80e1-acf4cabbafc5","Type":"ContainerStarted","Data":"b4c407f376327e99b61bd7c535ed0f054ed56a02162eac24d990515a2ce42142"} Jul 28 15:35:13.423614 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.423531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" event={"ID":"64fee8a1-9410-4b99-80e1-acf4cabbafc5","Type":"ContainerStarted","Data":"0497d6fc5a635313024ac33624021d0da8c5cad4905aa99bc2e9b2dcc55c49ac"} Jul 28 15:35:13.424712 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:13.424695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"e53803bfd6408b4ac2cbe0e86af2c695563a80dd1703e5eb7e5a70e6f9b394ec"} Jul 28 15:35:14.432102 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:14.432060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" event={"ID":"64fee8a1-9410-4b99-80e1-acf4cabbafc5","Type":"ContainerStarted","Data":"eccbc55aeacaf64b64e461f48d6e8651f04e5d91247c2d64ee8c8180b61558fb"} Jul 28 15:35:14.434303 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:14.434278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8b2k" event={"ID":"a31d5bef-2243-4d1e-8f68-a39f99cbe67f","Type":"ContainerStarted","Data":"bd96ca06b4a75738ec12fe2ce73f9611549dcf2b59a730ef0111984d6385a1d8"} Jul 28 15:35:14.434442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:14.434327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8b2k" event={"ID":"a31d5bef-2243-4d1e-8f68-a39f99cbe67f","Type":"ContainerStarted","Data":"2633b40835b6535dd672c8de147e1338e74bc93afc7392859d5c4e730f4a9787"} Jul 28 15:35:14.452755 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:14.452707 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f89mg" podStartSLOduration=1.923357036 podStartE2EDuration="3.452691757s" totalImagesPullingTime="1.529334721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:14.450102253 +0000 UTC m=+101.131159279" watchObservedRunningTime="2026-07-28 15:35:14.452691757 +0000 UTC m=+101.133748769" Jul 28 15:35:14.469261 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:14.469204 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t8b2k" podStartSLOduration=1.6912371099999999 podStartE2EDuration="3.469191792s" totalImagesPullingTime="1.656586394s" totalInitContainerRuntime="121.368288ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:14.467517792 +0000 UTC m=+101.148574805" watchObservedRunningTime="2026-07-28 15:35:14.469191792 +0000 UTC m=+101.150248804" Jul 28 15:35:15.440503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:15.440469 2580 generic.go:356] "Generic (PLEG): container finished" podID="0ae083da-da86-4cf5-a7ac-fec1dcf6704d" containerID="612add7c7b958c787ed6ef51c8bf60b2f484485b888424c5d4392090a9b747cc" exitCode=0 Jul 28 15:35:15.440928 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:15.440557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerDied","Data":"612add7c7b958c787ed6ef51c8bf60b2f484485b888424c5d4392090a9b747cc"} Jul 28 15:35:15.860720 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:15.860650 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:15.901285 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:15.901257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:15.901431 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:15.901353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084554 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084981 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fssb\" (UniqueName: \"kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084981 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.084981 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.084892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.185984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.185949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fssb\" (UniqueName: \"kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.186453 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.187171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.186898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.187171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.187008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.187171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.187111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.187171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.187145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.189108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.189084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.189760 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.189738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.196943 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.196921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fssb\" (UniqueName: \"kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb\") pod \"console-5578fdfb76-wv4bp\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.210620 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.210592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:16.401181 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.401072 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:16.403872 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:35:16.403777 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c71b7e_3439_433f_97f6_56a021c356ac.slice/crio-dcaf575360cc1ce9cd0b8091589e7c2ede815a9c044681b14808e627542c264a WatchSource:0}: Error finding container dcaf575360cc1ce9cd0b8091589e7c2ede815a9c044681b14808e627542c264a: Status 404 returned error can't find the container with id dcaf575360cc1ce9cd0b8091589e7c2ede815a9c044681b14808e627542c264a Jul 28 15:35:16.447075 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:16.446941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578fdfb76-wv4bp" event={"ID":"24c71b7e-3439-433f-97f6-56a021c356ac","Type":"ContainerStarted","Data":"dcaf575360cc1ce9cd0b8091589e7c2ede815a9c044681b14808e627542c264a"} Jul 28 15:35:17.452399 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:17.452373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578fdfb76-wv4bp" event={"ID":"24c71b7e-3439-433f-97f6-56a021c356ac","Type":"ContainerStarted","Data":"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70"} Jul 28 15:35:17.499217 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:17.499173 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5578fdfb76-wv4bp" podStartSLOduration=2.499157722 podStartE2EDuration="2.499157722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:17.49657868 +0000 UTC m=+104.177635715" watchObservedRunningTime="2026-07-28 15:35:17.499157722 +0000 UTC m=+104.180214733" Jul 28 15:35:18.461711 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:18.461676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"1aed4b979b3b4c6d9ca7c424bc8cc48600804f400f194301d7fab841ce6d4167"} Jul 28 15:35:18.462112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:18.461764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"ab32f6bc21102faa7451b6460b48c490b2a5a17bc00ba15bf0efc33a139926d4"} Jul 28 15:35:18.462112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:18.461826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"7ba2b7d4b82246999e88a79189f60577e271309ac9adc3fa53e07407f8d89101"} Jul 28 15:35:18.462112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:18.461864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"33d17cd1df13eb5e97e855526ab43ae68602d9652f18cc8f9be03669e344abfc"} Jul 28 15:35:18.462112 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:18.461893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"5ba602b8f960442f5a9f344525f14ae6a1ed219d1f80596c5174183ded8fdf94"} Jul 28 15:35:19.469713 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:19.469679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ae083da-da86-4cf5-a7ac-fec1dcf6704d","Type":"ContainerStarted","Data":"1bdbaa4187f4ddd5212b4429dc6843dbc1bb496a818d53614a25937cac345ffe"} Jul 28 15:35:19.503494 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:19.503405 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.484219361 podStartE2EDuration="7.503389996s" totalImagesPullingTime="5.010211482s" totalInitContainerRuntime="8.959153ms" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:19.499595814 +0000 UTC m=+106.180652838" watchObservedRunningTime="2026-07-28 15:35:19.503389996 +0000 UTC m=+106.184447009" Jul 28 15:35:23.745639 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.745608 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:23.771957 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.771927 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:35:23.869924 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.869887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:35:23.870100 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.870005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949614 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949614 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949853 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949853 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4xh\" (UniqueName: \"kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949853 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949853 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:23.949853 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:23.949751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4xh\" (UniqueName: \"kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051306 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051306 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051405 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051520 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051945 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051945 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.051945 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.051972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.052396 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.052378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.053784 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.053762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.053990 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.053972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.063189 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.063164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4xh\" (UniqueName: \"kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh\") pod \"console-5698ff855-4xmkd\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.179458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.179415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:24.314585 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.314563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:35:24.317058 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:35:24.317031 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8dbd23_21a3_41fc_bd98_48f5da47f301.slice/crio-beb63e168bf1fda559b49e1b04b0514d294521da550dc53587f8c02a772bf917 WatchSource:0}: Error finding container beb63e168bf1fda559b49e1b04b0514d294521da550dc53587f8c02a772bf917: Status 404 returned error can't find the container with id beb63e168bf1fda559b49e1b04b0514d294521da550dc53587f8c02a772bf917 Jul 28 15:35:24.496273 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.496243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698ff855-4xmkd" event={"ID":"8b8dbd23-21a3-41fc-bd98-48f5da47f301","Type":"ContainerStarted","Data":"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4"} Jul 28 15:35:24.496409 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.496315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698ff855-4xmkd" event={"ID":"8b8dbd23-21a3-41fc-bd98-48f5da47f301","Type":"ContainerStarted","Data":"beb63e168bf1fda559b49e1b04b0514d294521da550dc53587f8c02a772bf917"} Jul 28 15:35:24.520061 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:24.520019 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5698ff855-4xmkd" podStartSLOduration=1.520004146 podStartE2EDuration="1.520004146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:24.518571545 +0000 UTC m=+111.199628557" watchObservedRunningTime="2026-07-28 15:35:24.520004146 +0000 UTC m=+111.201061157" Jul 28 15:35:26.210851 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:26.210811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:32.528839 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:32.528772 2580 generic.go:356] "Generic (PLEG): container finished" podID="ad57ff0f-57a8-4509-9a41-329fa0b76aa4" containerID="9ad8310994e855a5293aea28e6bf6ea6b1de227c29b7201fcb622165b4ac03a5" exitCode=0 Jul 28 15:35:32.529284 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:32.528835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" event={"ID":"ad57ff0f-57a8-4509-9a41-329fa0b76aa4","Type":"ContainerDied","Data":"9ad8310994e855a5293aea28e6bf6ea6b1de227c29b7201fcb622165b4ac03a5"} Jul 28 15:35:32.529284 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:32.529229 2580 scope.go:117] "RemoveContainer" containerID="9ad8310994e855a5293aea28e6bf6ea6b1de227c29b7201fcb622165b4ac03a5" Jul 28 15:35:33.535546 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:33.535512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgx6t" event={"ID":"ad57ff0f-57a8-4509-9a41-329fa0b76aa4","Type":"ContainerStarted","Data":"158517755e705d61c3293980315b0aeaf5afc305bc646eed48a640dd4497f070"} Jul 28 15:35:34.180191 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.180151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:34.180399 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.180219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:34.185177 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.185147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:34.543155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.543079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:35:34.593921 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.593882 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:35:34.705773 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.705729 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" podUID="99d0f269-fecd-4743-9e85-1618f741ccae" containerName="registry" containerID="cri-o://764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1" gracePeriod=30 Jul 28 15:35:34.954991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:34.954967 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:35:35.044737 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044737 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044993 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044993 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044993 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7c9c\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044993 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044918 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.044993 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.045261 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.044998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets\") pod \"99d0f269-fecd-4743-9e85-1618f741ccae\" (UID: \"99d0f269-fecd-4743-9e85-1618f741ccae\") " Jul 28 15:35:35.045539 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.045413 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:35.045539 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.045503 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:35.047486 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.047461 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:35:35.047585 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.047547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c" (OuterVolumeSpecName: "kube-api-access-p7c9c") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "kube-api-access-p7c9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:35:35.047664 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.047645 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:35:35.048015 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.047990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:35.048073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.048039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:35.053411 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.053387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "99d0f269-fecd-4743-9e85-1618f741ccae" (UID: "99d0f269-fecd-4743-9e85-1618f741ccae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146218 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-image-registry-private-configuration\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146252 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-registry-certificates\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146266 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7c9c\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-kube-api-access-p7c9c\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146277 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-registry-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146290 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99d0f269-fecd-4743-9e85-1618f741ccae-trusted-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146301 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99d0f269-fecd-4743-9e85-1618f741ccae-installation-pull-secrets\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146313 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99d0f269-fecd-4743-9e85-1618f741ccae-bound-sa-token\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.146671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.146325 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99d0f269-fecd-4743-9e85-1618f741ccae-ca-trust-extracted\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:35.545628 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.545590 2580 generic.go:356] "Generic (PLEG): container finished" podID="99d0f269-fecd-4743-9e85-1618f741ccae" containerID="764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1" exitCode=0 Jul 28 15:35:35.546068 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.545635 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" Jul 28 15:35:35.546068 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.545678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" event={"ID":"99d0f269-fecd-4743-9e85-1618f741ccae","Type":"ContainerDied","Data":"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1"} Jul 28 15:35:35.546068 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.545731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bff99df44-qlw9d" event={"ID":"99d0f269-fecd-4743-9e85-1618f741ccae","Type":"ContainerDied","Data":"3c055abf1c99375d2435d4c6623c2a045e88842fd86e630bee8abe1a79d2e1be"} Jul 28 15:35:35.546068 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.545757 2580 scope.go:117] "RemoveContainer" containerID="764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1" Jul 28 15:35:35.555253 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.555238 2580 scope.go:117] "RemoveContainer" containerID="764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1" Jul 28 15:35:35.555518 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:35:35.555491 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1\": container with ID starting with 764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1 not found: ID does not exist" containerID="764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1" Jul 28 15:35:35.555586 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.555528 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1"} err="failed to get container status \"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1\": rpc error: code = NotFound desc = could not find container \"764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1\": container with ID starting with 764a72183585555f20851833a6e096eca5230b934149f22264f6b741be7ef4f1 not found: ID does not exist" Jul 28 15:35:35.568713 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.568690 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:35:35.571336 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.571295 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bff99df44-qlw9d"] Jul 28 15:35:35.858155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:35.858078 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d0f269-fecd-4743-9e85-1618f741ccae" path="/var/lib/kubelet/pods/99d0f269-fecd-4743-9e85-1618f741ccae/volumes" Jul 28 15:35:47.596992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:47.596955 2580 generic.go:356] "Generic (PLEG): container finished" podID="398680a4-2b5a-4ee2-9a9f-3ec51e80043c" containerID="fc4c66e64af5e7a46e696d9c73662c74f7404c7be608f6f6173b1a9dcd3cf13b" exitCode=0 Jul 28 15:35:47.597452 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:47.597016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" event={"ID":"398680a4-2b5a-4ee2-9a9f-3ec51e80043c","Type":"ContainerDied","Data":"fc4c66e64af5e7a46e696d9c73662c74f7404c7be608f6f6173b1a9dcd3cf13b"} Jul 28 15:35:47.597452 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:47.597295 2580 scope.go:117] "RemoveContainer" containerID="fc4c66e64af5e7a46e696d9c73662c74f7404c7be608f6f6173b1a9dcd3cf13b" Jul 28 15:35:48.602478 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:48.602445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9" event={"ID":"398680a4-2b5a-4ee2-9a9f-3ec51e80043c","Type":"ContainerStarted","Data":"75e9ebe574713c80eb4f55234eedc2147b1be85aecd057d4493ab2bcb0252fcc"} Jul 28 15:35:48.765947 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:48.765891 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5578fdfb76-wv4bp" podUID="24c71b7e-3439-433f-97f6-56a021c356ac" containerName="console" containerID="cri-o://27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70" gracePeriod=15 Jul 28 15:35:49.007790 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.007767 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:49.060072 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060042 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060093 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fssb\" (UniqueName: \"kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060172 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060327 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060222 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060327 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060270 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060327 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config\") pod \"24c71b7e-3439-433f-97f6-56a021c356ac\" (UID: \"24c71b7e-3439-433f-97f6-56a021c356ac\") " Jul 28 15:35:49.060522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:49.060675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060655 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.060768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.060727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca" (OuterVolumeSpecName: "service-ca") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:49.061033 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.061006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config" (OuterVolumeSpecName: "console-config") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:49.061128 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.061108 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:49.062510 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.062482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:49.062589 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.062514 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:49.062589 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.062563 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb" (OuterVolumeSpecName: "kube-api-access-6fssb") pod "24c71b7e-3439-433f-97f6-56a021c356ac" (UID: "24c71b7e-3439-433f-97f6-56a021c356ac"). InnerVolumeSpecName "kube-api-access-6fssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:35:49.161832 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161788 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.161832 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161830 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fssb\" (UniqueName: \"kubernetes.io/projected/24c71b7e-3439-433f-97f6-56a021c356ac-kube-api-access-6fssb\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.161999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161841 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.161999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161850 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c71b7e-3439-433f-97f6-56a021c356ac-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.161999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161858 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.161999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.161866 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c71b7e-3439-433f-97f6-56a021c356ac-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:35:49.607222 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.607134 2580 generic.go:356] "Generic (PLEG): container finished" podID="24c71b7e-3439-433f-97f6-56a021c356ac" containerID="27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70" exitCode=0 Jul 28 15:35:49.607222 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.607179 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578fdfb76-wv4bp" Jul 28 15:35:49.607675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.607230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578fdfb76-wv4bp" event={"ID":"24c71b7e-3439-433f-97f6-56a021c356ac","Type":"ContainerDied","Data":"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70"} Jul 28 15:35:49.607675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.607278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578fdfb76-wv4bp" event={"ID":"24c71b7e-3439-433f-97f6-56a021c356ac","Type":"ContainerDied","Data":"dcaf575360cc1ce9cd0b8091589e7c2ede815a9c044681b14808e627542c264a"} Jul 28 15:35:49.607675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.607299 2580 scope.go:117] "RemoveContainer" containerID="27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70" Jul 28 15:35:49.616300 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.616285 2580 scope.go:117] "RemoveContainer" containerID="27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70" Jul 28 15:35:49.616549 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:35:49.616530 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70\": container with ID starting with 27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70 not found: ID does not exist" containerID="27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70" Jul 28 15:35:49.616604 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.616563 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70"} err="failed to get container status \"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70\": rpc error: code = NotFound desc = could not find container \"27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70\": container with ID starting with 27532d79fc63d5c0a4901f2f95f8087d8f16ac2f9d351da7f08d027af1618e70 not found: ID does not exist" Jul 28 15:35:49.631185 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.631155 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:49.639906 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.639881 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5578fdfb76-wv4bp"] Jul 28 15:35:49.858313 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:49.858236 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c71b7e-3439-433f-97f6-56a021c356ac" path="/var/lib/kubelet/pods/24c71b7e-3439-433f-97f6-56a021c356ac/volumes" Jul 28 15:35:53.625399 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:53.625367 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3909d0f-91ee-4bf5-ada5-bfc4f4362102" containerID="c1874137fcef420dba093ec89877afdd99a99bcab85b507aac7342107f3bcc58" exitCode=0 Jul 28 15:35:53.625820 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:53.625441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" event={"ID":"c3909d0f-91ee-4bf5-ada5-bfc4f4362102","Type":"ContainerDied","Data":"c1874137fcef420dba093ec89877afdd99a99bcab85b507aac7342107f3bcc58"} Jul 28 15:35:53.625820 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:53.625765 2580 scope.go:117] "RemoveContainer" containerID="c1874137fcef420dba093ec89877afdd99a99bcab85b507aac7342107f3bcc58" Jul 28 15:35:54.630728 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:54.630696 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4rqn4" event={"ID":"c3909d0f-91ee-4bf5-ada5-bfc4f4362102","Type":"ContainerStarted","Data":"aab4c81415ccc545bb66b8b59cbf2650ce761a7703ecb98f74216a2ac27c8d0d"} Jul 28 15:35:59.618553 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.618490 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dc5db556d-djsgh" podUID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" containerName="console" containerID="cri-o://f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b" gracePeriod=15 Jul 28 15:35:59.866465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.866435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:35:59.955072 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955036 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvlsb\" (UniqueName: \"kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955193 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955247 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955510 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955265 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config\") pod \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\" (UID: \"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a\") " Jul 28 15:35:59.955647 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955565 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:59.955647 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:59.955840 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955650 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config" (OuterVolumeSpecName: "console-config") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:59.955840 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.955769 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:35:59.957528 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.957507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:59.958036 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.958010 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:35:59.958123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:35:59.958036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb" (OuterVolumeSpecName: "kube-api-access-kvlsb") pod "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" (UID: "7edcd7bb-b8fd-4ec8-ada4-da7e370a112a"). InnerVolumeSpecName "kube-api-access-kvlsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:36:00.056180 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056140 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvlsb\" (UniqueName: \"kubernetes.io/projected/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-kube-api-access-kvlsb\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056180 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056173 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056180 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056182 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056180 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056190 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056424 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056198 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056424 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056208 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.056424 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.056217 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:36:00.655759 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.655726 2580 generic.go:356] "Generic (PLEG): container finished" podID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" containerID="f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b" exitCode=0 Jul 28 15:36:00.656162 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.655774 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc5db556d-djsgh" Jul 28 15:36:00.656162 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.655841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc5db556d-djsgh" event={"ID":"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a","Type":"ContainerDied","Data":"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b"} Jul 28 15:36:00.656162 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.655894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc5db556d-djsgh" event={"ID":"7edcd7bb-b8fd-4ec8-ada4-da7e370a112a","Type":"ContainerDied","Data":"8ec17af0bb429c9a7b2a6ef85a27a2f1031a91ec46f215b5e2545d81177b91f8"} Jul 28 15:36:00.656162 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.655921 2580 scope.go:117] "RemoveContainer" containerID="f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b" Jul 28 15:36:00.664781 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.664766 2580 scope.go:117] "RemoveContainer" containerID="f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b" Jul 28 15:36:00.665077 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:36:00.665059 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b\": container with ID starting with f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b not found: ID does not exist" containerID="f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b" Jul 28 15:36:00.665117 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.665086 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b"} err="failed to get container status \"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b\": rpc error: code = NotFound desc = could not find container \"f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b\": container with ID starting with f5db2841a96b98ffcef0033f88032272c1f4da23aad641b5ee70fe1e1c8afb9b not found: ID does not exist" Jul 28 15:36:00.680345 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.680321 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:36:00.684925 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:00.684907 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc5db556d-djsgh"] Jul 28 15:36:01.860172 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:01.860134 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" path="/var/lib/kubelet/pods/7edcd7bb-b8fd-4ec8-ada4-da7e370a112a/volumes" Jul 28 15:36:27.422241 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422160 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422599 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c71b7e-3439-433f-97f6-56a021c356ac" containerName="console" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422618 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c71b7e-3439-433f-97f6-56a021c356ac" containerName="console" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422643 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d0f269-fecd-4743-9e85-1618f741ccae" containerName="registry" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422653 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d0f269-fecd-4743-9e85-1618f741ccae" containerName="registry" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422666 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" containerName="console" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422676 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" containerName="console" Jul 28 15:36:27.422751 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422749 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d0f269-fecd-4743-9e85-1618f741ccae" containerName="registry" Jul 28 15:36:27.423149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422762 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c71b7e-3439-433f-97f6-56a021c356ac" containerName="console" Jul 28 15:36:27.423149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.422770 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edcd7bb-b8fd-4ec8-ada4-da7e370a112a" containerName="console" Jul 28 15:36:27.425314 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.425294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.439159 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.439135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:36:27.465277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465443 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqpq\" (UniqueName: \"kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465634 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.465634 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.465493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566620 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.566850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqpq\" (UniqueName: \"kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.567114 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.566981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.567547 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.567522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.567733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.567706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.567869 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.567670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.567869 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.567855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.569340 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.569322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.569437 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.569420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.576110 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.576086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqpq\" (UniqueName: \"kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq\") pod \"console-6f5b8766cf-mj5h5\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.736263 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.736168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:27.871029 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:27.871007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:36:27.873470 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:36:27.873443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82d83dc_60ea_4c35_9e0c_585558ee6d79.slice/crio-f194838b715526e17dbb851a700ac79eae382ac73eaad51bb43908bfc9f69692 WatchSource:0}: Error finding container f194838b715526e17dbb851a700ac79eae382ac73eaad51bb43908bfc9f69692: Status 404 returned error can't find the container with id f194838b715526e17dbb851a700ac79eae382ac73eaad51bb43908bfc9f69692 Jul 28 15:36:28.774646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:28.774611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5b8766cf-mj5h5" event={"ID":"b82d83dc-60ea-4c35-9e0c-585558ee6d79","Type":"ContainerStarted","Data":"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d"} Jul 28 15:36:28.775077 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:28.774667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5b8766cf-mj5h5" event={"ID":"b82d83dc-60ea-4c35-9e0c-585558ee6d79","Type":"ContainerStarted","Data":"f194838b715526e17dbb851a700ac79eae382ac73eaad51bb43908bfc9f69692"} Jul 28 15:36:28.795503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:28.795452 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5b8766cf-mj5h5" podStartSLOduration=1.7954357010000002 podStartE2EDuration="1.795435701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:36:28.793779949 +0000 UTC m=+175.474836961" watchObservedRunningTime="2026-07-28 15:36:28.795435701 +0000 UTC m=+175.476492713" Jul 28 15:36:37.736749 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:37.736712 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:37.737172 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:37.736767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:37.741658 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:37.741637 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:37.818242 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:37.818214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:36:37.877495 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:36:37.877459 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:37:02.899757 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:02.899713 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5698ff855-4xmkd" podUID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" containerName="console" containerID="cri-o://90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4" gracePeriod=15 Jul 28 15:37:03.146642 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.146620 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:37:03.266973 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.266941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.266973 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.266973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267217 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267016 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267217 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267217 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk4xh\" (UniqueName: \"kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267217 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267425 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert\") pod \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\" (UID: \"8b8dbd23-21a3-41fc-bd98-48f5da47f301\") " Jul 28 15:37:03.267623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca" (OuterVolumeSpecName: "service-ca") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:37:03.267623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267614 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config" (OuterVolumeSpecName: "console-config") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:37:03.267782 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267726 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:37:03.267861 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.267763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:37:03.269301 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.269271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:37:03.269486 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.269463 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh" (OuterVolumeSpecName: "kube-api-access-sk4xh") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "kube-api-access-sk4xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:37:03.269546 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.269525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8b8dbd23-21a3-41fc-bd98-48f5da47f301" (UID: "8b8dbd23-21a3-41fc-bd98-48f5da47f301"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:37:03.368016 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.367979 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368016 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368007 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368016 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368017 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368026 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368035 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4xh\" (UniqueName: \"kubernetes.io/projected/8b8dbd23-21a3-41fc-bd98-48f5da47f301-kube-api-access-sk4xh\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368045 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8dbd23-21a3-41fc-bd98-48f5da47f301-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.368223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.368053 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b8dbd23-21a3-41fc-bd98-48f5da47f301-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:37:03.916438 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.916407 2580 generic.go:356] "Generic (PLEG): container finished" podID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" containerID="90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4" exitCode=0 Jul 28 15:37:03.916831 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.916456 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698ff855-4xmkd" Jul 28 15:37:03.916831 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.916445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698ff855-4xmkd" event={"ID":"8b8dbd23-21a3-41fc-bd98-48f5da47f301","Type":"ContainerDied","Data":"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4"} Jul 28 15:37:03.916831 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.916569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698ff855-4xmkd" event={"ID":"8b8dbd23-21a3-41fc-bd98-48f5da47f301","Type":"ContainerDied","Data":"beb63e168bf1fda559b49e1b04b0514d294521da550dc53587f8c02a772bf917"} Jul 28 15:37:03.916831 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.916591 2580 scope.go:117] "RemoveContainer" containerID="90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4" Jul 28 15:37:03.926772 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.926750 2580 scope.go:117] "RemoveContainer" containerID="90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4" Jul 28 15:37:03.927092 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:37:03.927073 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4\": container with ID starting with 90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4 not found: ID does not exist" containerID="90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4" Jul 28 15:37:03.927153 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.927101 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4"} err="failed to get container status \"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4\": rpc error: code = NotFound desc = could not find container \"90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4\": container with ID starting with 90e5dd6f320cec48fb8409cbbab85baa43c89ca81cf08231b63742800ca453c4 not found: ID does not exist" Jul 28 15:37:03.936145 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.936123 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:37:03.939404 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:03.939385 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5698ff855-4xmkd"] Jul 28 15:37:05.858234 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:37:05.858187 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" path="/var/lib/kubelet/pods/8b8dbd23-21a3-41fc-bd98-48f5da47f301/volumes" Jul 28 15:38:33.721387 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:33.721355 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:38:33.724139 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:33.724109 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:38:33.728416 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:33.728394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:38:33.731494 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:33.731477 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:38:33.731887 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:33.731862 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 15:38:48.648686 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.648644 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj"] Jul 28 15:38:48.650942 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.648997 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" containerName="console" Jul 28 15:38:48.650942 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.649013 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" containerName="console" Jul 28 15:38:48.650942 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.649080 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8dbd23-21a3-41fc-bd98-48f5da47f301" containerName="console" Jul 28 15:38:48.651873 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.651858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.656138 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.656109 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x8dlj\"" Jul 28 15:38:48.656266 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.656170 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 15:38:48.656266 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.656171 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 15:38:48.670505 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.670477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj"] Jul 28 15:38:48.673435 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.673406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq5r\" (UniqueName: \"kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.673575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.673473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.673575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.673523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.774766 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.774720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq5r\" (UniqueName: \"kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.774974 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.774788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.774974 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.774885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.775211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.775190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.775264 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.775248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.784550 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.784518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq5r\" (UniqueName: \"kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:48.960759 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:48.960728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:38:49.094322 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:49.094294 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj"] Jul 28 15:38:49.097986 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:38:49.097955 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f89270_cfa7_47ed_ba80_e0f6907999b9.slice/crio-32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf WatchSource:0}: Error finding container 32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf: Status 404 returned error can't find the container with id 32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf Jul 28 15:38:49.100177 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:49.100163 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:38:49.321475 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:49.321391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" event={"ID":"c3f89270-cfa7-47ed-ba80-e0f6907999b9","Type":"ContainerStarted","Data":"32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf"} Jul 28 15:38:56.353522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:56.353483 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerID="1690512467f2055d80db81a650341187687967f7f384cff34fe835b6060e1ae8" exitCode=0 Jul 28 15:38:56.353991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:56.353574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" event={"ID":"c3f89270-cfa7-47ed-ba80-e0f6907999b9","Type":"ContainerDied","Data":"1690512467f2055d80db81a650341187687967f7f384cff34fe835b6060e1ae8"} Jul 28 15:38:59.366238 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:59.366205 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerID="3e4c4c95db21c66485846d6ad1fabcdb5210655f673985a31b4974219db00faa" exitCode=0 Jul 28 15:38:59.366618 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:38:59.366290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" event={"ID":"c3f89270-cfa7-47ed-ba80-e0f6907999b9","Type":"ContainerDied","Data":"3e4c4c95db21c66485846d6ad1fabcdb5210655f673985a31b4974219db00faa"} Jul 28 15:39:06.398761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:06.398728 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerID="9558b3b1049e3cce72149ae8bee68a8b3ed292f8a6d21bd0a78575d3bf6b3d0a" exitCode=0 Jul 28 15:39:06.399150 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:06.398785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" event={"ID":"c3f89270-cfa7-47ed-ba80-e0f6907999b9","Type":"ContainerDied","Data":"9558b3b1049e3cce72149ae8bee68a8b3ed292f8a6d21bd0a78575d3bf6b3d0a"} Jul 28 15:39:07.525405 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.525382 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:39:07.645836 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.645778 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvq5r\" (UniqueName: \"kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r\") pod \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " Jul 28 15:39:07.646025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.645888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util\") pod \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " Jul 28 15:39:07.646025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.645934 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle\") pod \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\" (UID: \"c3f89270-cfa7-47ed-ba80-e0f6907999b9\") " Jul 28 15:39:07.646583 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.646556 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle" (OuterVolumeSpecName: "bundle") pod "c3f89270-cfa7-47ed-ba80-e0f6907999b9" (UID: "c3f89270-cfa7-47ed-ba80-e0f6907999b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:39:07.648184 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.648164 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r" (OuterVolumeSpecName: "kube-api-access-fvq5r") pod "c3f89270-cfa7-47ed-ba80-e0f6907999b9" (UID: "c3f89270-cfa7-47ed-ba80-e0f6907999b9"). InnerVolumeSpecName "kube-api-access-fvq5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:39:07.650184 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.650160 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util" (OuterVolumeSpecName: "util") pod "c3f89270-cfa7-47ed-ba80-e0f6907999b9" (UID: "c3f89270-cfa7-47ed-ba80-e0f6907999b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:39:07.746561 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.746535 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:39:07.746561 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.746559 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3f89270-cfa7-47ed-ba80-e0f6907999b9-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:39:07.746561 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:07.746567 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq5r\" (UniqueName: \"kubernetes.io/projected/c3f89270-cfa7-47ed-ba80-e0f6907999b9-kube-api-access-fvq5r\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:39:08.408444 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:08.408415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" event={"ID":"c3f89270-cfa7-47ed-ba80-e0f6907999b9","Type":"ContainerDied","Data":"32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf"} Jul 28 15:39:08.408616 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:08.408459 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32262e67f6c6c1a0f05a83b8de404e5df6386d3cf0a4e190f2768cef50a0aadf" Jul 28 15:39:08.408616 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:08.408461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5jvdlj" Jul 28 15:39:10.650275 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650243 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx"] Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650541 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="extract" Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650553 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="extract" Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650562 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="pull" Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650567 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="pull" Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650577 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="util" Jul 28 15:39:10.650627 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650582 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="util" Jul 28 15:39:10.650821 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.650633 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f89270-cfa7-47ed-ba80-e0f6907999b9" containerName="extract" Jul 28 15:39:10.691321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.691283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx"] Jul 28 15:39:10.691449 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.691378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.696522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.696498 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-lzvgp\"" Jul 28 15:39:10.696671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.696533 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 28 15:39:10.696671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.696511 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 28 15:39:10.696671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.696498 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 28 15:39:10.772155 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.772117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztcq\" (UniqueName: \"kubernetes.io/projected/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-kube-api-access-vztcq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.772318 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.772181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.872992 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.872960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztcq\" (UniqueName: \"kubernetes.io/projected/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-kube-api-access-vztcq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.873205 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.873022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.875817 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.875777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:10.884923 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:10.884887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztcq\" (UniqueName: \"kubernetes.io/projected/b1f04358-c3d6-4345-a3a2-5c6a47c17bfe-kube-api-access-vztcq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-47hgx\" (UID: \"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:11.001105 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:11.001067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:11.163485 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:11.163462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx"] Jul 28 15:39:11.165884 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:39:11.165856 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f04358_c3d6_4345_a3a2_5c6a47c17bfe.slice/crio-ed8bbebd1d02557f4d59bc17e26e96c20eb6940cdf5a2655aef0aefaa2a43275 WatchSource:0}: Error finding container ed8bbebd1d02557f4d59bc17e26e96c20eb6940cdf5a2655aef0aefaa2a43275: Status 404 returned error can't find the container with id ed8bbebd1d02557f4d59bc17e26e96c20eb6940cdf5a2655aef0aefaa2a43275 Jul 28 15:39:11.421244 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:11.421208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" event={"ID":"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe","Type":"ContainerStarted","Data":"ed8bbebd1d02557f4d59bc17e26e96c20eb6940cdf5a2655aef0aefaa2a43275"} Jul 28 15:39:15.385529 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.385488 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-jgww5"] Jul 28 15:39:15.407978 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.407948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-jgww5"] Jul 28 15:39:15.408131 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.408037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.410925 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.410905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 28 15:39:15.411102 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.410977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xxg4f\"" Jul 28 15:39:15.411179 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.411164 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 28 15:39:15.442140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.442109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" event={"ID":"b1f04358-c3d6-4345-a3a2-5c6a47c17bfe","Type":"ContainerStarted","Data":"dfc922d96c67f3ef962d4a12be3340fd5a9e2b1401ec82cf005b6d5bae31daf4"} Jul 28 15:39:15.442276 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.442254 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:15.462372 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.462333 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" podStartSLOduration=2.153144159 podStartE2EDuration="5.462319231s" totalImagesPullingTime="3.309175072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:39:15.461044128 +0000 UTC m=+342.142101174" watchObservedRunningTime="2026-07-28 15:39:15.462319231 +0000 UTC m=+342.143376242" Jul 28 15:39:15.511620 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.511587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.511785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.511652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/22638594-8e44-4ab8-8fd0-4dcca6de7782-cabundle0\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.511785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.511739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzd9p\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-kube-api-access-dzd9p\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.601561 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.601528 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp"] Jul 28 15:39:15.613208 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.613174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzd9p\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-kube-api-access-dzd9p\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.613353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.613304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.613457 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.613437 2580 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Jul 28 15:39:15.613457 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.613459 2580 secret.go:281] references non-existent secret key: ca.crt Jul 28 15:39:15.613650 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.613470 2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 28 15:39:15.613650 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.613485 2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-jgww5: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 28 15:39:15.613650 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.613558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/22638594-8e44-4ab8-8fd0-4dcca6de7782-cabundle0\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.613650 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.613629 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates podName:22638594-8e44-4ab8-8fd0-4dcca6de7782 nodeName:}" failed. No retries permitted until 2026-07-28 15:39:16.113610528 +0000 UTC m=+342.794667519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates") pod "keda-operator-77f648d694-jgww5" (UID: "22638594-8e44-4ab8-8fd0-4dcca6de7782") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 28 15:39:15.614177 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.614156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/22638594-8e44-4ab8-8fd0-4dcca6de7782-cabundle0\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.618635 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.618608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp"] Jul 28 15:39:15.618736 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.618723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.621504 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.621484 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 28 15:39:15.625573 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.625326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzd9p\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-kube-api-access-dzd9p\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:15.714530 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.714497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fd71717-4848-4742-94fa-79f5fa9e690d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.714695 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.714538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.714695 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.714615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ssr\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-kube-api-access-d4ssr\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.816006 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.815967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fd71717-4848-4742-94fa-79f5fa9e690d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.816201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.816015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.816201 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.816134 2580 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Jul 28 15:39:15.816201 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.816157 2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp: secret "keda-metrics-apiserver-certs" not found Jul 28 15:39:15.816337 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:39:15.816214 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates podName:7fd71717-4848-4742-94fa-79f5fa9e690d nodeName:}" failed. No retries permitted until 2026-07-28 15:39:16.316194894 +0000 UTC m=+342.997251898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates") pod "keda-metrics-apiserver-55557df8c-p4qqp" (UID: "7fd71717-4848-4742-94fa-79f5fa9e690d") : secret "keda-metrics-apiserver-certs" not found Jul 28 15:39:15.816337 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.816151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ssr\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-kube-api-access-d4ssr\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.816411 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.816367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fd71717-4848-4742-94fa-79f5fa9e690d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:15.827420 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:15.827394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ssr\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-kube-api-access-d4ssr\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:16.119084 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.119006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:16.121507 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.121477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22638594-8e44-4ab8-8fd0-4dcca6de7782-certificates\") pod \"keda-operator-77f648d694-jgww5\" (UID: \"22638594-8e44-4ab8-8fd0-4dcca6de7782\") " pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:16.317922 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.317889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:16.320813 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.320774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:16.323537 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.323513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fd71717-4848-4742-94fa-79f5fa9e690d-certificates\") pod \"keda-metrics-apiserver-55557df8c-p4qqp\" (UID: \"7fd71717-4848-4742-94fa-79f5fa9e690d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:16.452108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.452078 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-jgww5"] Jul 28 15:39:16.455493 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:39:16.455471 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22638594_8e44_4ab8_8fd0_4dcca6de7782.slice/crio-e0881cd78fff4054d14392fd7f1b2544604b995bad857831be50153311e901c4 WatchSource:0}: Error finding container e0881cd78fff4054d14392fd7f1b2544604b995bad857831be50153311e901c4: Status 404 returned error can't find the container with id e0881cd78fff4054d14392fd7f1b2544604b995bad857831be50153311e901c4 Jul 28 15:39:16.536288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.536254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:16.673512 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:16.673486 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp"] Jul 28 15:39:16.675554 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:39:16.675530 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd71717_4848_4742_94fa_79f5fa9e690d.slice/crio-2ffbb75526f0b735486dacf3d36e254335b56798a8168489ba9e2e63c798c4a2 WatchSource:0}: Error finding container 2ffbb75526f0b735486dacf3d36e254335b56798a8168489ba9e2e63c798c4a2: Status 404 returned error can't find the container with id 2ffbb75526f0b735486dacf3d36e254335b56798a8168489ba9e2e63c798c4a2 Jul 28 15:39:17.454141 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:17.454097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-jgww5" event={"ID":"22638594-8e44-4ab8-8fd0-4dcca6de7782","Type":"ContainerStarted","Data":"e0881cd78fff4054d14392fd7f1b2544604b995bad857831be50153311e901c4"} Jul 28 15:39:17.455404 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:17.455374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" event={"ID":"7fd71717-4848-4742-94fa-79f5fa9e690d","Type":"ContainerStarted","Data":"2ffbb75526f0b735486dacf3d36e254335b56798a8168489ba9e2e63c798c4a2"} Jul 28 15:39:20.473167 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.473126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" event={"ID":"7fd71717-4848-4742-94fa-79f5fa9e690d","Type":"ContainerStarted","Data":"90b3cd563616ecf29c059011c3fafac96e33260b86f358f133e5cdfd12672647"} Jul 28 15:39:20.473671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.473305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:20.474850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.474825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-jgww5" event={"ID":"22638594-8e44-4ab8-8fd0-4dcca6de7782","Type":"ContainerStarted","Data":"c211c6a75be9dd1277683f2eb6c6b152f4590b1b54a33ad4d77e332a3a0086b1"} Jul 28 15:39:20.474995 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.474973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:39:20.492248 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.492184 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" podStartSLOduration=1.7900596370000001 podStartE2EDuration="5.49216342s" totalImagesPullingTime="3.702103783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:39:20.489604276 +0000 UTC m=+347.170661289" watchObservedRunningTime="2026-07-28 15:39:20.49216342 +0000 UTC m=+347.173220432" Jul 28 15:39:20.510433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:20.510387 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-jgww5" podStartSLOduration=1.581470694 podStartE2EDuration="5.510366408s" totalImagesPullingTime="3.928895714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:39:20.508178851 +0000 UTC m=+347.189235857" watchObservedRunningTime="2026-07-28 15:39:20.510366408 +0000 UTC m=+347.191423420" Jul 28 15:39:31.484136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:31.484056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-p4qqp" Jul 28 15:39:36.449717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:36.449688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-47hgx" Jul 28 15:39:41.482056 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:39:41.482020 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-jgww5" Jul 28 15:40:31.440588 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.440550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:40:31.443200 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.443185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.445868 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.445846 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6tflh\"" Jul 28 15:40:31.445868 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.445846 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 15:40:31.447136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.447123 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 15:40:31.447193 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.447143 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 28 15:40:31.453015 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.452990 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:40:31.471062 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.471038 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-xft2l"] Jul 28 15:40:31.473561 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.473543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.476570 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.476552 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 15:40:31.476679 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.476562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gwzjp\"" Jul 28 15:40:31.481465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.481445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xft2l"] Jul 28 15:40:31.540192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.540157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49td\" (UniqueName: \"kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.540369 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.540209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.641655 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.641614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.641847 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.641669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrh6\" (UniqueName: \"kubernetes.io/projected/03971e83-2e78-4725-ab35-e345215cc1e2-kube-api-access-rzrh6\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.641847 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.641702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03971e83-2e78-4725-ab35-e345215cc1e2-data\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.641847 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.641753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49td\" (UniqueName: \"kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.641847 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:40:31.641778 2580 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 28 15:40:31.641990 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:40:31.641897 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert podName:f90cd54a-739c-4dd7-851c-a944267cebcd nodeName:}" failed. No retries permitted until 2026-07-28 15:40:32.141880177 +0000 UTC m=+418.822937167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert") pod "kserve-controller-manager-85d8b8ddfc-kvff8" (UID: "f90cd54a-739c-4dd7-851c-a944267cebcd") : secret "kserve-webhook-server-cert" not found Jul 28 15:40:31.650912 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.650888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49td\" (UniqueName: \"kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:31.742862 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.742748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03971e83-2e78-4725-ab35-e345215cc1e2-data\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.743023 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.742870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrh6\" (UniqueName: \"kubernetes.io/projected/03971e83-2e78-4725-ab35-e345215cc1e2-kube-api-access-rzrh6\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.743151 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.743132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03971e83-2e78-4725-ab35-e345215cc1e2-data\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.752244 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.752221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrh6\" (UniqueName: \"kubernetes.io/projected/03971e83-2e78-4725-ab35-e345215cc1e2-kube-api-access-rzrh6\") pod \"seaweedfs-64568bcd49-xft2l\" (UID: \"03971e83-2e78-4725-ab35-e345215cc1e2\") " pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.783870 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.783835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:31.911674 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:31.911650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xft2l"] Jul 28 15:40:31.914079 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:40:31.914042 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03971e83_2e78_4725_ab35_e345215cc1e2.slice/crio-7d7c4dd55315a50392ed8fa826437dd55d5b35d47ebd757e1d627d8fd430cc66 WatchSource:0}: Error finding container 7d7c4dd55315a50392ed8fa826437dd55d5b35d47ebd757e1d627d8fd430cc66: Status 404 returned error can't find the container with id 7d7c4dd55315a50392ed8fa826437dd55d5b35d47ebd757e1d627d8fd430cc66 Jul 28 15:40:32.147078 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.146996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:32.149464 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.149446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") pod \"kserve-controller-manager-85d8b8ddfc-kvff8\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:32.354863 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.354828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:32.509009 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.508981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:40:32.548694 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:40:32.548665 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90cd54a_739c_4dd7_851c_a944267cebcd.slice/crio-397e12e2907609b07bc228233558e8c479d0e0624e3de7ca4e96c07494253571 WatchSource:0}: Error finding container 397e12e2907609b07bc228233558e8c479d0e0624e3de7ca4e96c07494253571: Status 404 returned error can't find the container with id 397e12e2907609b07bc228233558e8c479d0e0624e3de7ca4e96c07494253571 Jul 28 15:40:32.798267 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.798174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xft2l" event={"ID":"03971e83-2e78-4725-ab35-e345215cc1e2","Type":"ContainerStarted","Data":"7d7c4dd55315a50392ed8fa826437dd55d5b35d47ebd757e1d627d8fd430cc66"} Jul 28 15:40:32.799304 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:32.799284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" event={"ID":"f90cd54a-739c-4dd7-851c-a944267cebcd","Type":"ContainerStarted","Data":"397e12e2907609b07bc228233558e8c479d0e0624e3de7ca4e96c07494253571"} Jul 28 15:40:36.410830 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.410775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 15:40:36.826008 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.825915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xft2l" event={"ID":"03971e83-2e78-4725-ab35-e345215cc1e2","Type":"ContainerStarted","Data":"8a79f8a05c73e0466c0f8a1719dd2fee67375a38159248eba2e271e7e4ecc7e1"} Jul 28 15:40:36.826008 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.825981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:36.827293 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.827271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" event={"ID":"f90cd54a-739c-4dd7-851c-a944267cebcd","Type":"ContainerStarted","Data":"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6"} Jul 28 15:40:36.827406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.827391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:40:36.843329 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.843285 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-xft2l" podStartSLOduration=1.3509189560000001 podStartE2EDuration="5.843270641s" totalImagesPullingTime="4.492351685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:36.840912207 +0000 UTC m=+423.521969219" watchObservedRunningTime="2026-07-28 15:40:36.843270641 +0000 UTC m=+423.524327653" Jul 28 15:40:36.859897 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:36.859854 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" podStartSLOduration=2.113500794 podStartE2EDuration="5.859839562s" totalImagesPullingTime="3.746338768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:36.858775817 +0000 UTC m=+423.539832830" watchObservedRunningTime="2026-07-28 15:40:36.859839562 +0000 UTC m=+423.540896578" Jul 28 15:40:42.834405 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:42.834374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-xft2l" Jul 28 15:40:46.083181 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:40:46.083149 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:41:07.837128 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:07.837054 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:41:11.103667 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.103601 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5b8766cf-mj5h5" podUID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" containerName="console" containerID="cri-o://6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d" gracePeriod=15 Jul 28 15:41:11.356063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.356002 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:41:11.489358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489460 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489541 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkqpq\" (UniqueName: \"kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq\") pod \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\" (UID: \"b82d83dc-60ea-4c35-9e0c-585558ee6d79\") " Jul 28 15:41:11.489878 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489841 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca" (OuterVolumeSpecName: "service-ca") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:11.489878 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config" (OuterVolumeSpecName: "console-config") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:11.489967 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.489878 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:11.490260 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.490195 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:41:11.491818 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.491772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:41:11.491902 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.491818 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:41:11.491902 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.491834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq" (OuterVolumeSpecName: "kube-api-access-fkqpq") pod "b82d83dc-60ea-4c35-9e0c-585558ee6d79" (UID: "b82d83dc-60ea-4c35-9e0c-585558ee6d79"). InnerVolumeSpecName "kube-api-access-fkqpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:41:11.590523 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590489 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590523 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590517 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqpq\" (UniqueName: \"kubernetes.io/projected/b82d83dc-60ea-4c35-9e0c-585558ee6d79-kube-api-access-fkqpq\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590532 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590548 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590560 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590571 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b82d83dc-60ea-4c35-9e0c-585558ee6d79-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.590717 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.590594 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82d83dc-60ea-4c35-9e0c-585558ee6d79-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:11.981225 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.981197 2580 generic.go:356] "Generic (PLEG): container finished" podID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" containerID="6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d" exitCode=0 Jul 28 15:41:11.981360 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.981242 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5b8766cf-mj5h5" Jul 28 15:41:11.981360 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.981253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5b8766cf-mj5h5" event={"ID":"b82d83dc-60ea-4c35-9e0c-585558ee6d79","Type":"ContainerDied","Data":"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d"} Jul 28 15:41:11.981360 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.981291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5b8766cf-mj5h5" event={"ID":"b82d83dc-60ea-4c35-9e0c-585558ee6d79","Type":"ContainerDied","Data":"f194838b715526e17dbb851a700ac79eae382ac73eaad51bb43908bfc9f69692"} Jul 28 15:41:11.981360 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.981310 2580 scope.go:117] "RemoveContainer" containerID="6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d" Jul 28 15:41:11.989999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.989978 2580 scope.go:117] "RemoveContainer" containerID="6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d" Jul 28 15:41:11.990255 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:41:11.990236 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d\": container with ID starting with 6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d not found: ID does not exist" containerID="6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d" Jul 28 15:41:11.990319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.990264 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d"} err="failed to get container status \"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d\": rpc error: code = NotFound desc = could not find container \"6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d\": container with ID starting with 6e5dbc1bdae5697ba82dfad334c52d83b7a481583bd966d02d4e3afd9fe5088d not found: ID does not exist" Jul 28 15:41:11.999005 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:11.998981 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:41:12.004806 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:12.004776 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5b8766cf-mj5h5"] Jul 28 15:41:13.858363 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:13.858330 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" path="/var/lib/kubelet/pods/b82d83dc-60ea-4c35-9e0c-585558ee6d79/volumes" Jul 28 15:41:14.221115 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.221081 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:41:14.221347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.221320 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" podUID="f90cd54a-739c-4dd7-851c-a944267cebcd" containerName="manager" containerID="cri-o://5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6" gracePeriod=10 Jul 28 15:41:14.242059 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.242030 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-jpk7t"] Jul 28 15:41:14.242493 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.242475 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" containerName="console" Jul 28 15:41:14.242493 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.242494 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" containerName="console" Jul 28 15:41:14.242649 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.242599 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82d83dc-60ea-4c35-9e0c-585558ee6d79" containerName="console" Jul 28 15:41:14.244880 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.244860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.257714 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.257575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-jpk7t"] Jul 28 15:41:14.416351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.416319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxjt\" (UniqueName: \"kubernetes.io/projected/cb267dde-5e85-483c-a2ed-b7b52a141b0c-kube-api-access-gtxjt\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.416478 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.416362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb267dde-5e85-483c-a2ed-b7b52a141b0c-cert\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.462988 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.462964 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:41:14.517244 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.517161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxjt\" (UniqueName: \"kubernetes.io/projected/cb267dde-5e85-483c-a2ed-b7b52a141b0c-kube-api-access-gtxjt\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.517244 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.517202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb267dde-5e85-483c-a2ed-b7b52a141b0c-cert\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.519651 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.519628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb267dde-5e85-483c-a2ed-b7b52a141b0c-cert\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.525494 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.525472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxjt\" (UniqueName: \"kubernetes.io/projected/cb267dde-5e85-483c-a2ed-b7b52a141b0c-kube-api-access-gtxjt\") pod \"kserve-controller-manager-85d8b8ddfc-jpk7t\" (UID: \"cb267dde-5e85-483c-a2ed-b7b52a141b0c\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.601506 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.601474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:14.617850 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.617818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") pod \"f90cd54a-739c-4dd7-851c-a944267cebcd\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " Jul 28 15:41:14.618022 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.617885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d49td\" (UniqueName: \"kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td\") pod \"f90cd54a-739c-4dd7-851c-a944267cebcd\" (UID: \"f90cd54a-739c-4dd7-851c-a944267cebcd\") " Jul 28 15:41:14.620025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.619999 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert" (OuterVolumeSpecName: "cert") pod "f90cd54a-739c-4dd7-851c-a944267cebcd" (UID: "f90cd54a-739c-4dd7-851c-a944267cebcd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:41:14.620153 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.620032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td" (OuterVolumeSpecName: "kube-api-access-d49td") pod "f90cd54a-739c-4dd7-851c-a944267cebcd" (UID: "f90cd54a-739c-4dd7-851c-a944267cebcd"). InnerVolumeSpecName "kube-api-access-d49td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:41:14.718482 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.718448 2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f90cd54a-739c-4dd7-851c-a944267cebcd-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:14.718482 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.718481 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49td\" (UniqueName: \"kubernetes.io/projected/f90cd54a-739c-4dd7-851c-a944267cebcd-kube-api-access-d49td\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:41:14.733924 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.733897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-jpk7t"] Jul 28 15:41:14.736229 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:41:14.736204 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb267dde_5e85_483c_a2ed_b7b52a141b0c.slice/crio-a29e6c1acfe32e22517037db1917d796675969285dd90e15794a463ba07c2096 WatchSource:0}: Error finding container a29e6c1acfe32e22517037db1917d796675969285dd90e15794a463ba07c2096: Status 404 returned error can't find the container with id a29e6c1acfe32e22517037db1917d796675969285dd90e15794a463ba07c2096 Jul 28 15:41:14.997320 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.997283 2580 generic.go:356] "Generic (PLEG): container finished" podID="f90cd54a-739c-4dd7-851c-a944267cebcd" containerID="5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6" exitCode=0 Jul 28 15:41:14.997780 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.997326 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" Jul 28 15:41:14.997780 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.997323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" event={"ID":"f90cd54a-739c-4dd7-851c-a944267cebcd","Type":"ContainerDied","Data":"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6"} Jul 28 15:41:14.997780 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.997371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-kvff8" event={"ID":"f90cd54a-739c-4dd7-851c-a944267cebcd","Type":"ContainerDied","Data":"397e12e2907609b07bc228233558e8c479d0e0624e3de7ca4e96c07494253571"} Jul 28 15:41:14.997780 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.997398 2580 scope.go:117] "RemoveContainer" containerID="5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6" Jul 28 15:41:14.998768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:14.998743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" event={"ID":"cb267dde-5e85-483c-a2ed-b7b52a141b0c","Type":"ContainerStarted","Data":"a29e6c1acfe32e22517037db1917d796675969285dd90e15794a463ba07c2096"} Jul 28 15:41:15.006139 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:15.006125 2580 scope.go:117] "RemoveContainer" containerID="5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6" Jul 28 15:41:15.006382 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:41:15.006367 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6\": container with ID starting with 5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6 not found: ID does not exist" containerID="5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6" Jul 28 15:41:15.006426 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:15.006390 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6"} err="failed to get container status \"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6\": rpc error: code = NotFound desc = could not find container \"5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6\": container with ID starting with 5314debaef7dc08482a0b46b8b42eabd46d67c92b6b3aba88d33fc2337dfdde6 not found: ID does not exist" Jul 28 15:41:15.020355 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:15.020329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:41:15.024085 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:15.024065 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-kvff8"] Jul 28 15:41:15.858304 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:15.858272 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90cd54a-739c-4dd7-851c-a944267cebcd" path="/var/lib/kubelet/pods/f90cd54a-739c-4dd7-851c-a944267cebcd/volumes" Jul 28 15:41:16.004860 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:16.004825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" event={"ID":"cb267dde-5e85-483c-a2ed-b7b52a141b0c","Type":"ContainerStarted","Data":"e6a3c12a9d8bf922d73c5a55757e8f0591103b9426d85ae916bc9eecc6e79311"} Jul 28 15:41:16.005222 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:16.004932 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:16.021455 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:16.021411 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" podStartSLOduration=1.680166273 podStartE2EDuration="2.021395162s" totalImagesPullingTime="341.228889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:16.020774697 +0000 UTC m=+462.701831721" watchObservedRunningTime="2026-07-28 15:41:16.021395162 +0000 UTC m=+462.702452173" Jul 28 15:41:47.015582 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:47.015548 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-jpk7t" Jul 28 15:41:48.550675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.550641 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pdnrk"] Jul 28 15:41:48.551063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.551001 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90cd54a-739c-4dd7-851c-a944267cebcd" containerName="manager" Jul 28 15:41:48.551063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.551014 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90cd54a-739c-4dd7-851c-a944267cebcd" containerName="manager" Jul 28 15:41:48.551137 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.551077 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90cd54a-739c-4dd7-851c-a944267cebcd" containerName="manager" Jul 28 15:41:48.553221 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.553198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:48.556163 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.556144 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 28 15:41:48.556352 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.556335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r22wt\"" Jul 28 15:41:48.563667 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.563647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pdnrk"] Jul 28 15:41:48.594331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.594290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmh7\" (UniqueName: \"kubernetes.io/projected/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-kube-api-access-9rmh7\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:48.594503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.594343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:48.695063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.695027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmh7\" (UniqueName: \"kubernetes.io/projected/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-kube-api-access-9rmh7\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:48.695240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.695078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:48.695240 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:41:48.695202 2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 28 15:41:48.695318 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:41:48.695266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert podName:ec5fbd9f-706c-4c94-a3cd-cb9fa831b382 nodeName:}" failed. No retries permitted until 2026-07-28 15:41:49.195249274 +0000 UTC m=+495.876306272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert") pod "odh-model-controller-75d9cdbddf-pdnrk" (UID: "ec5fbd9f-706c-4c94-a3cd-cb9fa831b382") : secret "odh-model-controller-webhook-cert" not found Jul 28 15:41:48.704449 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:48.704423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmh7\" (UniqueName: \"kubernetes.io/projected/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-kube-api-access-9rmh7\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:49.200475 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:49.200444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:49.203026 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:49.203001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5fbd9f-706c-4c94-a3cd-cb9fa831b382-cert\") pod \"odh-model-controller-75d9cdbddf-pdnrk\" (UID: \"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382\") " pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:49.464434 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:49.464337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:49.600251 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:49.600222 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pdnrk"] Jul 28 15:41:49.602434 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:41:49.602408 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5fbd9f_706c_4c94_a3cd_cb9fa831b382.slice/crio-219841b9c415ba899741d617af76e32a11be3e1c0d26d5cb31c5e38cf01ae926 WatchSource:0}: Error finding container 219841b9c415ba899741d617af76e32a11be3e1c0d26d5cb31c5e38cf01ae926: Status 404 returned error can't find the container with id 219841b9c415ba899741d617af76e32a11be3e1c0d26d5cb31c5e38cf01ae926 Jul 28 15:41:50.166492 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:50.166463 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" event={"ID":"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382","Type":"ContainerStarted","Data":"219841b9c415ba899741d617af76e32a11be3e1c0d26d5cb31c5e38cf01ae926"} Jul 28 15:41:53.187442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:53.187409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" event={"ID":"ec5fbd9f-706c-4c94-a3cd-cb9fa831b382","Type":"ContainerStarted","Data":"c088236a50dd15a8ef7a10c1ebdaf572454e502f8f70c4c54d8fddfa539aea34"} Jul 28 15:41:53.187845 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:53.187521 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:41:53.203950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:41:53.203903 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" podStartSLOduration=2.421177051 podStartE2EDuration="5.203887593s" totalImagesPullingTime="2.782710542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:53.202377911 +0000 UTC m=+499.883434924" watchObservedRunningTime="2026-07-28 15:41:53.203887593 +0000 UTC m=+499.884944605" Jul 28 15:42:04.194691 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:04.194665 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-pdnrk" Jul 28 15:42:05.072167 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.072131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7kls2"] Jul 28 15:42:05.074605 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.074584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7kls2" Jul 28 15:42:05.083152 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.083129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7kls2"] Jul 28 15:42:05.240864 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.240835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.241234 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.240871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kj7\" (UniqueName: \"kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.342070 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.341982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.342070 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.342022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kj7\" (UniqueName: \"kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.342424 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.342400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.352092 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.352068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kj7\" (UniqueName: \"kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7\") pod \"s3-init-7kls2\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " pod="kserve/s3-init-7kls2" Jul 28 15:42:05.395051 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.395023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7kls2" Jul 28 15:42:05.525468 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:05.525442 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7kls2"] Jul 28 15:42:05.527355 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:42:05.527327 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9ee021a_e5d1_4a48_934e_eab4b4efe2eb.slice/crio-2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5 WatchSource:0}: Error finding container 2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5: Status 404 returned error can't find the container with id 2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5 Jul 28 15:42:06.244653 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:06.244618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerStarted","Data":"2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5"} Jul 28 15:42:09.260123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:09.260089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerStarted","Data":"eca1a27f9af4ad654b5c8fc6b3e7ab0511072b99e79f6c0ec7f7e7f1c9db3998"} Jul 28 15:42:11.271833 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:11.271785 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerID="eca1a27f9af4ad654b5c8fc6b3e7ab0511072b99e79f6c0ec7f7e7f1c9db3998" exitCode=0 Jul 28 15:42:11.271833 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:11.271824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerDied","Data":"eca1a27f9af4ad654b5c8fc6b3e7ab0511072b99e79f6c0ec7f7e7f1c9db3998"} Jul 28 15:42:16.297654 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:16.297619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerStarted","Data":"27b9ab1edb8290a3ce4d47bc7d0ed4e635dc0984c17732ab67b196846d265464"} Jul 28 15:42:16.315391 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:42:16.315330 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-7kls2" podStartSLOduration=1.654782467 podStartE2EDuration="11.315314685s" totalImagesPullingTime="7.692893498s" totalInitContainerRuntime="1.96763872s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:42:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:42:16.314038192 +0000 UTC m=+522.995095206" watchObservedRunningTime="2026-07-28 15:42:16.315314685 +0000 UTC m=+522.996371697" Jul 28 15:43:21.596597 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:21.596554 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerID="27b9ab1edb8290a3ce4d47bc7d0ed4e635dc0984c17732ab67b196846d265464" exitCode=0 Jul 28 15:43:21.597144 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:21.596634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerDied","Data":"27b9ab1edb8290a3ce4d47bc7d0ed4e635dc0984c17732ab67b196846d265464"} Jul 28 15:43:22.732506 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.732478 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7kls2" Jul 28 15:43:22.787754 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.787720 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache\") pod \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " Jul 28 15:43:22.787957 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.787764 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6kj7\" (UniqueName: \"kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7\") pod \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\" (UID: \"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb\") " Jul 28 15:43:22.790002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.789976 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7" (OuterVolumeSpecName: "kube-api-access-r6kj7") pod "b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" (UID: "b9ee021a-e5d1-4a48-934e-eab4b4efe2eb"). InnerVolumeSpecName "kube-api-access-r6kj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:43:22.806712 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.806662 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" (UID: "b9ee021a-e5d1-4a48-934e-eab4b4efe2eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:43:22.888678 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.888584 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:43:22.888678 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:22.888618 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kj7\" (UniqueName: \"kubernetes.io/projected/b9ee021a-e5d1-4a48-934e-eab4b4efe2eb-kube-api-access-r6kj7\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:43:23.608140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:23.608102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7kls2" event={"ID":"b9ee021a-e5d1-4a48-934e-eab4b4efe2eb","Type":"ContainerDied","Data":"2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5"} Jul 28 15:43:23.608140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:23.608140 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7kls2" Jul 28 15:43:23.608140 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:23.608147 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2044f8c7c07d7b35031d04969e5dce18a3a92fa3e0eb5a71d876f043bf23a7e5" Jul 28 15:43:33.752158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:33.752129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:43:33.754990 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:33.754969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:43:33.757909 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:33.757883 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:43:33.760619 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:33.760589 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:43:37.197088 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197054 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:43:37.197462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197383 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerName="download-hf-model" Jul 28 15:43:37.197462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197396 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerName="download-hf-model" Jul 28 15:43:37.197462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197417 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerName="s3-init" Jul 28 15:43:37.197462 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197422 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerName="s3-init" Jul 28 15:43:37.197586 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.197477 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ee021a-e5d1-4a48-934e-eab4b4efe2eb" containerName="s3-init" Jul 28 15:43:37.200343 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.200323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.203684 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b15e5-kube-rbac-proxy-sar-config\"" Jul 28 15:43:37.203828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203716 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b15e5-predictor-serving-cert\"" Jul 28 15:43:37.203828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203735 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 15:43:37.203828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 15:43:37.203999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203824 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\"" Jul 28 15:43:37.203999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.203885 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 15:43:37.204206 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.204190 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 15:43:37.211440 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.211416 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:43:37.301292 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.301256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctl5h\" (UniqueName: \"kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.301292 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.301288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.301517 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.301400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.301517 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.301457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.301517 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.301474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402721 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctl5h\" (UniqueName: \"kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402721 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402721 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.402947 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.402927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.403202 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.403183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.403319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.403302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.405222 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.405195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.412214 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.412184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctl5h\" (UniqueName: \"kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h\") pod \"isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.511205 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.511100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:43:37.653270 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.653240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:43:37.655174 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:43:37.655142 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ad1ae0_75e7_4526_92a1_5221d00db5f7.slice/crio-a19b47f945fd784c2e35c54ae1abe6edf11d23e37cb2767b9a21d3ad802fc70e WatchSource:0}: Error finding container a19b47f945fd784c2e35c54ae1abe6edf11d23e37cb2767b9a21d3ad802fc70e: Status 404 returned error can't find the container with id a19b47f945fd784c2e35c54ae1abe6edf11d23e37cb2767b9a21d3ad802fc70e Jul 28 15:43:37.677416 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:37.677384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerStarted","Data":"a19b47f945fd784c2e35c54ae1abe6edf11d23e37cb2767b9a21d3ad802fc70e"} Jul 28 15:43:38.683319 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:38.683284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerStarted","Data":"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9"} Jul 28 15:43:41.701292 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:41.701256 2580 generic.go:356] "Generic (PLEG): container finished" podID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerID="a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9" exitCode=0 Jul 28 15:43:41.701753 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:41.701338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerDied","Data":"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9"} Jul 28 15:43:55.436486 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:55.436454 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:43:55.786206 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:55.786172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerStarted","Data":"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098"} Jul 28 15:43:58.804395 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:43:58.804359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerStarted","Data":"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61"} Jul 28 15:44:01.821439 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:01.821397 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerStarted","Data":"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90"} Jul 28 15:44:01.821948 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:01.821589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:44:01.821948 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:01.821739 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:44:01.823027 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:01.823000 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:01.844127 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:01.844079 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podStartSLOduration=2.125806544 podStartE2EDuration="24.844066525s" totalImagesPullingTime="18.817566966s" totalInitContainerRuntime="3.900693015s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:44:01.842573619 +0000 UTC m=+628.523630632" watchObservedRunningTime="2026-07-28 15:44:01.844066525 +0000 UTC m=+628.525123536" Jul 28 15:44:02.825745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:02.825709 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:44:02.826221 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:02.825869 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:02.827050 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:02.827024 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:02.829872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:02.829854 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:44:03.833307 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:03.833263 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:03.833703 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:03.833685 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:04.839133 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:04.839090 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:04.839611 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:04.839584 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:14.839874 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:14.839822 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:14.840291 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:14.840217 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:24.839920 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:24.839866 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:24.840323 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:24.840239 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:34.839611 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:34.839553 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:34.840192 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:34.840040 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:44.839971 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:44.839901 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:44.840427 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:44.840324 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:44:54.839468 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:54.839412 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:44:54.839916 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:44:54.839866 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:45:04.839154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:04.839106 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:45:04.839773 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:04.839746 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:45:14.839528 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:14.839497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:14.840457 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:14.840192 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:22.378182 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.378149 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:45:22.378786 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.378742 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" containerID="cri-o://a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098" gracePeriod=30 Jul 28 15:45:22.379133 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.378813 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" containerID="cri-o://2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90" gracePeriod=30 Jul 28 15:45:22.379133 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.378836 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" containerID="cri-o://25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61" gracePeriod=30 Jul 28 15:45:22.478641 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.478611 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:45:22.481307 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.481290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.484071 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.484043 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0be34-kube-rbac-proxy-sar-config\"" Jul 28 15:45:22.484366 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.484343 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0be34-predictor-serving-cert\"" Jul 28 15:45:22.493594 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.493570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:45:22.561229 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.561197 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:45:22.563916 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.563896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.566669 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.566647 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0be34-kube-rbac-proxy-sar-config\"" Jul 28 15:45:22.566761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.566674 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0be34-predictor-serving-cert\"" Jul 28 15:45:22.575178 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.575158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:45:22.578866 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.578842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.578975 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.578884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.578975 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.578954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zh5\" (UniqueName: \"kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.579085 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.578979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.579085 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.579017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.679816 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.679758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zh5\" (UniqueName: \"kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.679999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.679837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.679999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.679869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.679999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.679903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.679999 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.679951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.680159 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.680159 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:22.680033 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-serving-cert: secret "isvc-sklearn-graph-raw-0be34-predictor-serving-cert" not found Jul 28 15:45:22.680159 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zf9b\" (UniqueName: \"kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.680159 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:22.680118 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls podName:d162787f-50a7-4c70-9038-aae85237c21c nodeName:}" failed. No retries permitted until 2026-07-28 15:45:23.180098605 +0000 UTC m=+709.861155595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls") pod "isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" (UID: "d162787f-50a7-4c70-9038-aae85237c21c") : secret "isvc-sklearn-graph-raw-0be34-predictor-serving-cert" not found Jul 28 15:45:22.680288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.680288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.680288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.680481 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.680646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.680625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.681030 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.681013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.690776 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.690750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zh5\" (UniqueName: \"kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:22.781230 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zf9b\" (UniqueName: \"kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.781415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.781415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.781415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.781415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.781813 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.781781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.782041 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.782020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.782107 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.782034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.783744 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.783726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.790084 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.790066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zf9b\" (UniqueName: \"kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b\") pod \"isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:22.826264 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.826227 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:22.875076 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:22.875047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:23.010641 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.010619 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:45:23.012190 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:45:23.012158 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356cad_b50a_4a0b_a9a0_ffdc72d62681.slice/crio-0c55550d43128e296be05fdd8fc828e989b540b0c7155ca1032104fcf09aedbb WatchSource:0}: Error finding container 0c55550d43128e296be05fdd8fc828e989b540b0c7155ca1032104fcf09aedbb: Status 404 returned error can't find the container with id 0c55550d43128e296be05fdd8fc828e989b540b0c7155ca1032104fcf09aedbb Jul 28 15:45:23.184115 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.184085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:23.186556 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.186537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") pod \"isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:23.209766 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.209661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerStarted","Data":"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8"} Jul 28 15:45:23.209766 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.209745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerStarted","Data":"0c55550d43128e296be05fdd8fc828e989b540b0c7155ca1032104fcf09aedbb"} Jul 28 15:45:23.212347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.212318 2580 generic.go:356] "Generic (PLEG): container finished" podID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerID="25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61" exitCode=2 Jul 28 15:45:23.212480 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.212361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerDied","Data":"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61"} Jul 28 15:45:23.395977 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.395937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:23.533952 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:23.533919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:45:23.535376 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:45:23.535340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd162787f_50a7_4c70_9038_aae85237c21c.slice/crio-5864dad2796e2240d3507a9cb20613e64c54927d97b45cbe9f26e5ae1a90d5e1 WatchSource:0}: Error finding container 5864dad2796e2240d3507a9cb20613e64c54927d97b45cbe9f26e5ae1a90d5e1: Status 404 returned error can't find the container with id 5864dad2796e2240d3507a9cb20613e64c54927d97b45cbe9f26e5ae1a90d5e1 Jul 28 15:45:24.219202 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:24.219164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerStarted","Data":"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db"} Jul 28 15:45:24.219397 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:24.219265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerStarted","Data":"5864dad2796e2240d3507a9cb20613e64c54927d97b45cbe9f26e5ae1a90d5e1"} Jul 28 15:45:24.840290 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:24.840247 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:45:24.840710 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:24.840405 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:45:27.239348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:27.239269 2580 generic.go:356] "Generic (PLEG): container finished" podID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerID="a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098" exitCode=0 Jul 28 15:45:27.239734 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:27.239340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerDied","Data":"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098"} Jul 28 15:45:27.240945 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:27.240921 2580 generic.go:356] "Generic (PLEG): container finished" podID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerID="353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8" exitCode=0 Jul 28 15:45:27.241080 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:27.240944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerDied","Data":"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8"} Jul 28 15:45:27.826228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:27.826186 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:28.249068 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:28.249031 2580 generic.go:356] "Generic (PLEG): container finished" podID="d162787f-50a7-4c70-9038-aae85237c21c" containerID="c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db" exitCode=0 Jul 28 15:45:28.249557 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:28.249077 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerDied","Data":"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db"} Jul 28 15:45:30.264198 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.264163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerStarted","Data":"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f"} Jul 28 15:45:30.264646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.264262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerStarted","Data":"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3"} Jul 28 15:45:30.264646 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.264590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:30.264816 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.264774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:30.266064 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.266040 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:45:30.286856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:30.286772 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podStartSLOduration=4.636104888 podStartE2EDuration="8.28675506s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650650172s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:30.284234001 +0000 UTC m=+716.965291039" watchObservedRunningTime="2026-07-28 15:45:30.28675506 +0000 UTC m=+716.967812070" Jul 28 15:45:31.269686 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:31.269645 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:45:32.825967 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:32.825915 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:32.826348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:32.826061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:34.840187 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:34.840132 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:45:34.840571 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:34.840279 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:45:36.273854 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:36.273824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:45:36.274465 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:36.274428 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:45:37.826467 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:37.826420 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:42.826563 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:42.826519 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:44.839833 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:44.839778 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:45:44.840189 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:44.839930 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:44.840321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:44.840298 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Jul 28 15:45:44.840421 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:44.840389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:45.357694 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.357659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerStarted","Data":"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b"} Jul 28 15:45:45.357914 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.357732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerStarted","Data":"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f"} Jul 28 15:45:45.358059 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.358029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:45.358171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.358080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:45.359298 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.359274 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:45:45.378508 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:45.378460 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podStartSLOduration=1.8857543570000002 podStartE2EDuration="23.37844679s" totalImagesPullingTime="17.492567476s" totalInitContainerRuntime="4.000124957s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:45.376193712 +0000 UTC m=+732.057250736" watchObservedRunningTime="2026-07-28 15:45:45.37844679 +0000 UTC m=+732.059503800" Jul 28 15:45:46.275193 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:46.275156 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:45:46.362358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:46.362322 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:45:47.826756 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:47.826714 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:51.366777 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:51.366747 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:45:51.367444 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:51.367414 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:45:52.826657 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:52.826618 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Jul 28 15:45:53.027120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.027096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:53.148856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.148756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctl5h\" (UniqueName: \"kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h\") pod \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " Jul 28 15:45:53.148856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.148838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert\") pod \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " Jul 28 15:45:53.149058 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.148911 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls\") pod \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " Jul 28 15:45:53.149058 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.148944 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config\") pod \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " Jul 28 15:45:53.149058 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.149007 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location\") pod \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\" (UID: \"53ad1ae0-75e7-4526-92a1-5221d00db5f7\") " Jul 28 15:45:53.149262 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.149232 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53ad1ae0-75e7-4526-92a1-5221d00db5f7" (UID: "53ad1ae0-75e7-4526-92a1-5221d00db5f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:45:53.149357 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.149335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53ad1ae0-75e7-4526-92a1-5221d00db5f7" (UID: "53ad1ae0-75e7-4526-92a1-5221d00db5f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:45:53.149406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.149363 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53ad1ae0-75e7-4526-92a1-5221d00db5f7" (UID: "53ad1ae0-75e7-4526-92a1-5221d00db5f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:45:53.151070 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.151044 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h" (OuterVolumeSpecName: "kube-api-access-ctl5h") pod "53ad1ae0-75e7-4526-92a1-5221d00db5f7" (UID: "53ad1ae0-75e7-4526-92a1-5221d00db5f7"). InnerVolumeSpecName "kube-api-access-ctl5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:45:53.151193 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.151176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53ad1ae0-75e7-4526-92a1-5221d00db5f7" (UID: "53ad1ae0-75e7-4526-92a1-5221d00db5f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:45:53.250211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.250177 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ad1ae0-75e7-4526-92a1-5221d00db5f7-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:45:53.250211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.250207 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:45:53.250211 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.250217 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:45:53.250431 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.250227 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctl5h\" (UniqueName: \"kubernetes.io/projected/53ad1ae0-75e7-4526-92a1-5221d00db5f7-kube-api-access-ctl5h\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:45:53.250431 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.250236 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53ad1ae0-75e7-4526-92a1-5221d00db5f7-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:45:53.395196 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.395157 2580 generic.go:356] "Generic (PLEG): container finished" podID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerID="2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90" exitCode=0 Jul 28 15:45:53.395383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.395228 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" Jul 28 15:45:53.395383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.395246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerDied","Data":"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90"} Jul 28 15:45:53.395383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.395335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8" event={"ID":"53ad1ae0-75e7-4526-92a1-5221d00db5f7","Type":"ContainerDied","Data":"a19b47f945fd784c2e35c54ae1abe6edf11d23e37cb2767b9a21d3ad802fc70e"} Jul 28 15:45:53.395383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.395382 2580 scope.go:117] "RemoveContainer" containerID="2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90" Jul 28 15:45:53.404281 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.404266 2580 scope.go:117] "RemoveContainer" containerID="25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61" Jul 28 15:45:53.411986 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.411968 2580 scope.go:117] "RemoveContainer" containerID="a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098" Jul 28 15:45:53.418501 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.418472 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:45:53.420873 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.420861 2580 scope.go:117] "RemoveContainer" containerID="a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9" Jul 28 15:45:53.424880 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.422622 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8"] Jul 28 15:45:53.431341 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.431321 2580 scope.go:117] "RemoveContainer" containerID="2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90" Jul 28 15:45:53.431685 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:53.431662 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90\": container with ID starting with 2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90 not found: ID does not exist" containerID="2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90" Jul 28 15:45:53.431788 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.431692 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90"} err="failed to get container status \"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90\": rpc error: code = NotFound desc = could not find container \"2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90\": container with ID starting with 2bcac40a51dd5b782e61fb99876e4f3faae1f67392581a0b999cd21489182f90 not found: ID does not exist" Jul 28 15:45:53.431788 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.431710 2580 scope.go:117] "RemoveContainer" containerID="25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61" Jul 28 15:45:53.432046 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:53.432027 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61\": container with ID starting with 25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61 not found: ID does not exist" containerID="25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61" Jul 28 15:45:53.432093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.432052 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61"} err="failed to get container status \"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61\": rpc error: code = NotFound desc = could not find container \"25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61\": container with ID starting with 25747f199583cc0fa179f4edddaa524b26291f573b634545d0ab8fd65c95ca61 not found: ID does not exist" Jul 28 15:45:53.432093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.432069 2580 scope.go:117] "RemoveContainer" containerID="a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098" Jul 28 15:45:53.432341 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:53.432319 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098\": container with ID starting with a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098 not found: ID does not exist" containerID="a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098" Jul 28 15:45:53.432442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.432346 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098"} err="failed to get container status \"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098\": rpc error: code = NotFound desc = could not find container \"a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098\": container with ID starting with a3771a37af9d9b1aa3396af91ff2f309d0f052b9439046c810b846eafbea5098 not found: ID does not exist" Jul 28 15:45:53.432442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.432363 2580 scope.go:117] "RemoveContainer" containerID="a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9" Jul 28 15:45:53.432593 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:45:53.432576 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9\": container with ID starting with a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9 not found: ID does not exist" containerID="a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9" Jul 28 15:45:53.432643 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.432596 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9"} err="failed to get container status \"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9\": rpc error: code = NotFound desc = could not find container \"a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9\": container with ID starting with a041c57c34c1eaec51f59dfb49ab57a2f2cf6c30cdf0185d24c382f5f6fa59f9 not found: ID does not exist" Jul 28 15:45:53.864359 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:53.864329 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" path="/var/lib/kubelet/pods/53ad1ae0-75e7-4526-92a1-5221d00db5f7/volumes" Jul 28 15:45:56.274947 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:45:56.274909 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:01.367422 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:01.367382 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:46:06.275063 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:06.275025 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:11.367679 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:11.367635 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:46:16.275351 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:16.275314 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:21.367794 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:21.367754 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:46:26.274479 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:26.274436 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:31.367315 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:31.367268 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:46:36.274372 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:36.274331 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:36.853770 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:36.853732 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:46:41.368059 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:41.368018 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused" Jul 28 15:46:46.855436 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:46.855407 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:46:51.368973 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:46:51.368940 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:47:12.733813 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.733714 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:47:12.734228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.734032 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" containerID="cri-o://3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3" gracePeriod=30 Jul 28 15:47:12.734228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.734073 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kube-rbac-proxy" containerID="cri-o://5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f" gracePeriod=30 Jul 28 15:47:12.784879 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.784841 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:47:12.785233 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785220 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="storage-initializer" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785235 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="storage-initializer" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785246 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785251 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785262 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785267 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785275 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" Jul 28 15:47:12.785288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785280 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" Jul 28 15:47:12.785484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785335 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="agent" Jul 28 15:47:12.785484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kube-rbac-proxy" Jul 28 15:47:12.785484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.785349 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ad1ae0-75e7-4526-92a1-5221d00db5f7" containerName="kserve-container" Jul 28 15:47:12.787686 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.787670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:12.790471 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.790447 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c70d2-kube-rbac-proxy-sar-config\"" Jul 28 15:47:12.790607 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.790466 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c70d2-predictor-serving-cert\"" Jul 28 15:47:12.800228 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.800203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:47:12.916714 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.916679 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:47:12.916967 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.916938 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" containerID="cri-o://bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f" gracePeriod=30 Jul 28 15:47:12.917109 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.916984 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kube-rbac-proxy" containerID="cri-o://7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b" gracePeriod=30 Jul 28 15:47:12.920837 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.920765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:12.920957 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.920897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr7n\" (UniqueName: \"kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:12.920957 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.920937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:12.921073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.921018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:12.921073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:12.921065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.021605 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.021510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr7n\" (UniqueName: \"kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.021605 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.021568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.021843 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.021619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.021843 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.021675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.021843 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.021719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.022021 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:13.021999 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-c70d2-predictor-serving-cert" not found Jul 28 15:47:13.022072 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.022041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.022072 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:13.022066 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls podName:e6e2eb74-534c-4906-992f-d8ff1fabc0f4 nodeName:}" failed. No retries permitted until 2026-07-28 15:47:13.522045168 +0000 UTC m=+820.203102160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4") : secret "isvc-sklearn-graph-raw-hpa-c70d2-predictor-serving-cert" not found Jul 28 15:47:13.022411 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.022388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.022675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.022655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.030476 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.030457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr7n\" (UniqueName: \"kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.525668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.525633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.528087 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.528067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.699783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.699745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:13.780623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.780470 2580 generic.go:356] "Generic (PLEG): container finished" podID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerID="7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b" exitCode=2 Jul 28 15:47:13.781249 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.780511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerDied","Data":"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b"} Jul 28 15:47:13.783618 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.783589 2580 generic.go:356] "Generic (PLEG): container finished" podID="d162787f-50a7-4c70-9038-aae85237c21c" containerID="5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f" exitCode=2 Jul 28 15:47:13.783748 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.783718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerDied","Data":"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f"} Jul 28 15:47:13.836976 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:13.836886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:47:13.840542 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:47:13.840512 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e2eb74_534c_4906_992f_d8ff1fabc0f4.slice/crio-dc1a8a56dd6be19e4e7d349be695eb0fb6a1045463e99ca0cc55db9d28714bda WatchSource:0}: Error finding container dc1a8a56dd6be19e4e7d349be695eb0fb6a1045463e99ca0cc55db9d28714bda: Status 404 returned error can't find the container with id dc1a8a56dd6be19e4e7d349be695eb0fb6a1045463e99ca0cc55db9d28714bda Jul 28 15:47:14.789777 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:14.789740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerStarted","Data":"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e"} Jul 28 15:47:14.790205 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:14.789830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerStarted","Data":"dc1a8a56dd6be19e4e7d349be695eb0fb6a1045463e99ca0cc55db9d28714bda"} Jul 28 15:47:16.270559 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.270517 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused" Jul 28 15:47:16.363554 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.363518 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused" Jul 28 15:47:16.572954 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.572930 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:47:16.753461 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753435 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls\") pod \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " Jul 28 15:47:16.753621 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location\") pod \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " Jul 28 15:47:16.753621 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zf9b\" (UniqueName: \"kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b\") pod \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " Jul 28 15:47:16.753621 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753522 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config\") pod \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " Jul 28 15:47:16.753621 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753562 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert\") pod \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\" (UID: \"09356cad-b50a-4a0b-a9a0-ffdc72d62681\") " Jul 28 15:47:16.753773 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09356cad-b50a-4a0b-a9a0-ffdc72d62681" (UID: "09356cad-b50a-4a0b-a9a0-ffdc72d62681"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:47:16.753967 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.753941 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09356cad-b50a-4a0b-a9a0-ffdc72d62681" (UID: "09356cad-b50a-4a0b-a9a0-ffdc72d62681"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:47:16.754081 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.754009 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09356cad-b50a-4a0b-a9a0-ffdc72d62681" (UID: "09356cad-b50a-4a0b-a9a0-ffdc72d62681"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:47:16.755827 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.755783 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09356cad-b50a-4a0b-a9a0-ffdc72d62681" (UID: "09356cad-b50a-4a0b-a9a0-ffdc72d62681"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:47:16.755827 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.755794 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b" (OuterVolumeSpecName: "kube-api-access-5zf9b") pod "09356cad-b50a-4a0b-a9a0-ffdc72d62681" (UID: "09356cad-b50a-4a0b-a9a0-ffdc72d62681"). InnerVolumeSpecName "kube-api-access-5zf9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:47:16.801703 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.801670 2580 generic.go:356] "Generic (PLEG): container finished" podID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerID="bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f" exitCode=0 Jul 28 15:47:16.801875 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.801721 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" Jul 28 15:47:16.801875 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.801742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerDied","Data":"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f"} Jul 28 15:47:16.801875 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.801833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs" event={"ID":"09356cad-b50a-4a0b-a9a0-ffdc72d62681","Type":"ContainerDied","Data":"0c55550d43128e296be05fdd8fc828e989b540b0c7155ca1032104fcf09aedbb"} Jul 28 15:47:16.802092 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.801876 2580 scope.go:117] "RemoveContainer" containerID="7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b" Jul 28 15:47:16.810607 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.810590 2580 scope.go:117] "RemoveContainer" containerID="bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f" Jul 28 15:47:16.818047 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.818030 2580 scope.go:117] "RemoveContainer" containerID="353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8" Jul 28 15:47:16.825331 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.825311 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:47:16.826926 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.826422 2580 scope.go:117] "RemoveContainer" containerID="7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b" Jul 28 15:47:16.828219 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:16.828106 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b\": container with ID starting with 7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b not found: ID does not exist" containerID="7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b" Jul 28 15:47:16.828219 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.828142 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b"} err="failed to get container status \"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b\": rpc error: code = NotFound desc = could not find container \"7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b\": container with ID starting with 7d7af37739a9b582417d1face8ba7641c85cca36017501f0bc5aa3c501c6f15b not found: ID does not exist" Jul 28 15:47:16.828219 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.828164 2580 scope.go:117] "RemoveContainer" containerID="bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f" Jul 28 15:47:16.828625 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:16.828599 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f\": container with ID starting with bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f not found: ID does not exist" containerID="bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f" Jul 28 15:47:16.828625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.828632 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f"} err="failed to get container status \"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f\": rpc error: code = NotFound desc = could not find container \"bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f\": container with ID starting with bf3e2e852e08f3379e1cf31be2a174d4c7ee335d0782eb40179233ed7363620f not found: ID does not exist" Jul 28 15:47:16.828625 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.828686 2580 scope.go:117] "RemoveContainer" containerID="353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8" Jul 28 15:47:16.829506 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:16.829487 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8\": container with ID starting with 353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8 not found: ID does not exist" containerID="353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8" Jul 28 15:47:16.829587 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.829511 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8"} err="failed to get container status \"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8\": rpc error: code = NotFound desc = could not find container \"353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8\": container with ID starting with 353771e1602c3842f0f45b43b30682b92673a7bca3a894342e891c9aba0a9bc8 not found: ID does not exist" Jul 28 15:47:16.830883 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.830864 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs"] Jul 28 15:47:16.853970 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.853943 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused" Jul 28 15:47:16.854277 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.854259 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09356cad-b50a-4a0b-a9a0-ffdc72d62681-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:16.854349 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.854283 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:16.854349 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.854299 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zf9b\" (UniqueName: \"kubernetes.io/projected/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-api-access-5zf9b\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:16.854349 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.854312 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:16.854349 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:16.854325 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09356cad-b50a-4a0b-a9a0-ffdc72d62681-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:17.594121 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.594096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:47:17.760353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760277 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config\") pod \"d162787f-50a7-4c70-9038-aae85237c21c\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " Jul 28 15:47:17.760353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760326 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8zh5\" (UniqueName: \"kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5\") pod \"d162787f-50a7-4c70-9038-aae85237c21c\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " Jul 28 15:47:17.760353 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert\") pod \"d162787f-50a7-4c70-9038-aae85237c21c\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " Jul 28 15:47:17.760623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760382 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location\") pod \"d162787f-50a7-4c70-9038-aae85237c21c\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " Jul 28 15:47:17.760623 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") pod \"d162787f-50a7-4c70-9038-aae85237c21c\" (UID: \"d162787f-50a7-4c70-9038-aae85237c21c\") " Jul 28 15:47:17.760737 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760708 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d162787f-50a7-4c70-9038-aae85237c21c" (UID: "d162787f-50a7-4c70-9038-aae85237c21c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:47:17.760783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d162787f-50a7-4c70-9038-aae85237c21c" (UID: "d162787f-50a7-4c70-9038-aae85237c21c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:47:17.760783 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.760723 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d162787f-50a7-4c70-9038-aae85237c21c" (UID: "d162787f-50a7-4c70-9038-aae85237c21c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:47:17.762513 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.762489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5" (OuterVolumeSpecName: "kube-api-access-f8zh5") pod "d162787f-50a7-4c70-9038-aae85237c21c" (UID: "d162787f-50a7-4c70-9038-aae85237c21c"). InnerVolumeSpecName "kube-api-access-f8zh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:47:17.762687 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.762669 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d162787f-50a7-4c70-9038-aae85237c21c" (UID: "d162787f-50a7-4c70-9038-aae85237c21c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:47:17.809149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.809118 2580 generic.go:356] "Generic (PLEG): container finished" podID="d162787f-50a7-4c70-9038-aae85237c21c" containerID="3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3" exitCode=0 Jul 28 15:47:17.809273 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.809170 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" Jul 28 15:47:17.809273 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.809204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerDied","Data":"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3"} Jul 28 15:47:17.809273 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.809251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4" event={"ID":"d162787f-50a7-4c70-9038-aae85237c21c","Type":"ContainerDied","Data":"5864dad2796e2240d3507a9cb20613e64c54927d97b45cbe9f26e5ae1a90d5e1"} Jul 28 15:47:17.809413 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.809286 2580 scope.go:117] "RemoveContainer" containerID="5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f" Jul 28 15:47:17.810721 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.810699 2580 generic.go:356] "Generic (PLEG): container finished" podID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerID="46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e" exitCode=0 Jul 28 15:47:17.810825 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.810740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerDied","Data":"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e"} Jul 28 15:47:17.818739 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.818724 2580 scope.go:117] "RemoveContainer" containerID="3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3" Jul 28 15:47:17.827193 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.827176 2580 scope.go:117] "RemoveContainer" containerID="c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db" Jul 28 15:47:17.836257 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.836241 2580 scope.go:117] "RemoveContainer" containerID="5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f" Jul 28 15:47:17.836545 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:17.836527 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f\": container with ID starting with 5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f not found: ID does not exist" containerID="5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f" Jul 28 15:47:17.836611 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.836554 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f"} err="failed to get container status \"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f\": rpc error: code = NotFound desc = could not find container \"5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f\": container with ID starting with 5bf89c08b1c9b9e09589075eff8420936bedc320a268f2b43adda0d0a4b7d67f not found: ID does not exist" Jul 28 15:47:17.836611 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.836575 2580 scope.go:117] "RemoveContainer" containerID="3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3" Jul 28 15:47:17.836885 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:17.836867 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3\": container with ID starting with 3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3 not found: ID does not exist" containerID="3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3" Jul 28 15:47:17.836977 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.836893 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3"} err="failed to get container status \"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3\": rpc error: code = NotFound desc = could not find container \"3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3\": container with ID starting with 3c1a351b893b52f3297eed397e35f468645caed92c0b8252d96063537aaf90b3 not found: ID does not exist" Jul 28 15:47:17.836977 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.836913 2580 scope.go:117] "RemoveContainer" containerID="c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db" Jul 28 15:47:17.837229 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:47:17.837211 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db\": container with ID starting with c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db not found: ID does not exist" containerID="c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db" Jul 28 15:47:17.837278 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.837235 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db"} err="failed to get container status \"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db\": rpc error: code = NotFound desc = could not find container \"c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db\": container with ID starting with c2b633fb3ccaebf1465414acec624520a62a8560f7ca54067e3f3876343617db not found: ID does not exist" Jul 28 15:47:17.848562 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.848534 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:47:17.851272 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.851254 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4"] Jul 28 15:47:17.858541 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.858518 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" path="/var/lib/kubelet/pods/09356cad-b50a-4a0b-a9a0-ffdc72d62681/volumes" Jul 28 15:47:17.859058 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.859041 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d162787f-50a7-4c70-9038-aae85237c21c" path="/var/lib/kubelet/pods/d162787f-50a7-4c70-9038-aae85237c21c/volumes" Jul 28 15:47:17.861075 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.861055 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:17.861157 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.861075 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zh5\" (UniqueName: \"kubernetes.io/projected/d162787f-50a7-4c70-9038-aae85237c21c-kube-api-access-f8zh5\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:17.861157 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.861087 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d162787f-50a7-4c70-9038-aae85237c21c-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:17.861157 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.861095 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d162787f-50a7-4c70-9038-aae85237c21c-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:17.861157 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:17.861103 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d162787f-50a7-4c70-9038-aae85237c21c-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:47:18.818339 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.818305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerStarted","Data":"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df"} Jul 28 15:47:18.818693 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.818365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerStarted","Data":"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee"} Jul 28 15:47:18.818693 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.818629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:18.818810 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.818780 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:18.820171 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.820149 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:47:18.838631 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:18.838572 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podStartSLOduration=3.124168233 podStartE2EDuration="6.83855863s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714390397s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:47:18.83744333 +0000 UTC m=+825.518500341" watchObservedRunningTime="2026-07-28 15:47:18.83855863 +0000 UTC m=+825.519615640" Jul 28 15:47:19.823083 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:19.823040 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:47:24.827718 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:24.827670 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:47:24.828311 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:24.828119 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:47:34.828634 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:34.828598 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:47:44.829061 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:44.829020 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:47:54.828135 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:47:54.828093 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:48:04.828181 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:04.828139 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:48:14.828321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:14.828279 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:48:24.828215 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:24.828127 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Jul 28 15:48:31.858001 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:31.857975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:48:33.790682 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:33.790654 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:48:33.792925 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:33.792903 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:48:33.793379 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:33.793361 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:48:33.795704 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:48:33.795683 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:49:03.058358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:03.058320 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:49:03.059936 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:03.058543 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" containerID="cri-o://c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee" gracePeriod=30 Jul 28 15:49:03.059936 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:03.058596 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kube-rbac-proxy" containerID="cri-o://4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df" gracePeriod=30 Jul 28 15:49:03.309002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:03.308925 2580 generic.go:356] "Generic (PLEG): container finished" podID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerID="4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df" exitCode=2 Jul 28 15:49:03.309002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:03.308990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerDied","Data":"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df"} Jul 28 15:49:04.824208 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:04.824164 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused" Jul 28 15:49:07.805928 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.805898 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:49:07.977366 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977340 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config\") pod \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " Jul 28 15:49:07.977522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977384 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmr7n\" (UniqueName: \"kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n\") pod \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " Jul 28 15:49:07.977522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location\") pod \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " Jul 28 15:49:07.977522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977475 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") pod \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " Jul 28 15:49:07.977522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977509 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert\") pod \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\" (UID: \"e6e2eb74-534c-4906-992f-d8ff1fabc0f4\") " Jul 28 15:49:07.977862 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6e2eb74-534c-4906-992f-d8ff1fabc0f4" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:49:07.977862 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6e2eb74-534c-4906-992f-d8ff1fabc0f4" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:49:07.978005 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.977921 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6e2eb74-534c-4906-992f-d8ff1fabc0f4" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:49:07.979642 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.979620 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n" (OuterVolumeSpecName: "kube-api-access-fmr7n") pod "e6e2eb74-534c-4906-992f-d8ff1fabc0f4" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4"). InnerVolumeSpecName "kube-api-access-fmr7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:49:07.979741 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:07.979692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6e2eb74-534c-4906-992f-d8ff1fabc0f4" (UID: "e6e2eb74-534c-4906-992f-d8ff1fabc0f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:49:08.078419 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.078388 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:49:08.078419 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.078413 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmr7n\" (UniqueName: \"kubernetes.io/projected/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kube-api-access-fmr7n\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:49:08.078419 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.078422 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:49:08.078647 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.078433 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:49:08.078647 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.078443 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2eb74-534c-4906-992f-d8ff1fabc0f4-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:49:08.333108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.333011 2580 generic.go:356] "Generic (PLEG): container finished" podID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerID="c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee" exitCode=0 Jul 28 15:49:08.333108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.333069 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" Jul 28 15:49:08.333321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.333070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerDied","Data":"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee"} Jul 28 15:49:08.333321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.333192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq" event={"ID":"e6e2eb74-534c-4906-992f-d8ff1fabc0f4","Type":"ContainerDied","Data":"dc1a8a56dd6be19e4e7d349be695eb0fb6a1045463e99ca0cc55db9d28714bda"} Jul 28 15:49:08.333321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.333224 2580 scope.go:117] "RemoveContainer" containerID="4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df" Jul 28 15:49:08.341828 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.341789 2580 scope.go:117] "RemoveContainer" containerID="c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee" Jul 28 15:49:08.349525 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.349514 2580 scope.go:117] "RemoveContainer" containerID="46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e" Jul 28 15:49:08.357092 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.356996 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:49:08.357145 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357124 2580 scope.go:117] "RemoveContainer" containerID="4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df" Jul 28 15:49:08.357405 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:49:08.357374 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df\": container with ID starting with 4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df not found: ID does not exist" containerID="4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df" Jul 28 15:49:08.357444 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357411 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df"} err="failed to get container status \"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df\": rpc error: code = NotFound desc = could not find container \"4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df\": container with ID starting with 4b7fe48d0a20159163d43fb8ff35b9c5061abe83a5f24c984ac7aeffe1f0f7df not found: ID does not exist" Jul 28 15:49:08.357444 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357429 2580 scope.go:117] "RemoveContainer" containerID="c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee" Jul 28 15:49:08.357665 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:49:08.357650 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee\": container with ID starting with c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee not found: ID does not exist" containerID="c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee" Jul 28 15:49:08.357708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357670 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee"} err="failed to get container status \"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee\": rpc error: code = NotFound desc = could not find container \"c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee\": container with ID starting with c0a7af8fb81926eb570de776981eb2952dd14fcd9e3c14bddeff985a4404b8ee not found: ID does not exist" Jul 28 15:49:08.357708 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357695 2580 scope.go:117] "RemoveContainer" containerID="46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e" Jul 28 15:49:08.357952 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:49:08.357936 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e\": container with ID starting with 46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e not found: ID does not exist" containerID="46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e" Jul 28 15:49:08.358017 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.357956 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e"} err="failed to get container status \"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e\": rpc error: code = NotFound desc = could not find container \"46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e\": container with ID starting with 46b2cd6b273bd5ff71ed6d5084b6f1e8ef92f033c45e5cc9a8d24dda0b35c91e not found: ID does not exist" Jul 28 15:49:08.365747 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:08.365730 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq"] Jul 28 15:49:09.858004 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:09.857972 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" path="/var/lib/kubelet/pods/e6e2eb74-534c-4906-992f-d8ff1fabc0f4/volumes" Jul 28 15:49:23.154488 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154456 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154774 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="storage-initializer" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154784 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="storage-initializer" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154793 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154817 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154826 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kube-rbac-proxy" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154835 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kube-rbac-proxy" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154849 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="storage-initializer" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154854 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="storage-initializer" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154863 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kube-rbac-proxy" Jul 28 15:49:23.154871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154871 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kube-rbac-proxy" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154878 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154882 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kube-rbac-proxy" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154898 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kube-rbac-proxy" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154910 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154916 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154924 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="storage-initializer" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154928 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="storage-initializer" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154985 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154992 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kube-rbac-proxy" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.154997 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.155004 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d162787f-50a7-4c70-9038-aae85237c21c" containerName="kube-rbac-proxy" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.155010 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e2eb74-534c-4906-992f-d8ff1fabc0f4" containerName="kserve-container" Jul 28 15:49:23.155295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.155016 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09356cad-b50a-4a0b-a9a0-ffdc72d62681" containerName="kube-rbac-proxy" Jul 28 15:49:23.157415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.157400 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.160094 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160068 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-22083-kube-rbac-proxy-sar-config\"" Jul 28 15:49:23.160094 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 15:49:23.160281 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160186 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 15:49:23.160281 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160186 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 15:49:23.160281 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-22083-predictor-serving-cert\"" Jul 28 15:49:23.160547 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160507 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 15:49:23.160662 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.160646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\"" Jul 28 15:49:23.170622 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.170603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:49:23.193845 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.193789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjjz\" (UniqueName: \"kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.193954 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.193901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.193954 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.193931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.194047 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.193961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.194047 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.193994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295036 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjjz\" (UniqueName: \"kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295215 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295215 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295215 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295392 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:49:23.295232 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-serving-cert: secret "isvc-logger-raw-22083-predictor-serving-cert" not found Jul 28 15:49:23.295392 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295392 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:49:23.295316 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls podName:b5359347-8547-426c-a58a-7d8845de1e49 nodeName:}" failed. No retries permitted until 2026-07-28 15:49:23.795285724 +0000 UTC m=+950.476342715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls") pod "isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" (UID: "b5359347-8547-426c-a58a-7d8845de1e49") : secret "isvc-logger-raw-22083-predictor-serving-cert" not found Jul 28 15:49:23.295588 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295897 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295880 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.295991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.295973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.303726 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.303706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjjz\" (UniqueName: \"kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.799335 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.799295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:23.801894 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:23.801875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") pod \"isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:24.068269 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:24.068187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:24.206834 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:24.206775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:49:24.209791 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:49:24.209765 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5359347_8547_426c_a58a_7d8845de1e49.slice/crio-9394b095e76673614ad21df596113c67e9c76c7470e5b05a90b17022a40fe486 WatchSource:0}: Error finding container 9394b095e76673614ad21df596113c67e9c76c7470e5b05a90b17022a40fe486: Status 404 returned error can't find the container with id 9394b095e76673614ad21df596113c67e9c76c7470e5b05a90b17022a40fe486 Jul 28 15:49:24.410245 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:24.410209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerStarted","Data":"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4"} Jul 28 15:49:24.410427 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:24.410271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerStarted","Data":"9394b095e76673614ad21df596113c67e9c76c7470e5b05a90b17022a40fe486"} Jul 28 15:49:28.429349 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:28.429319 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5359347-8547-426c-a58a-7d8845de1e49" containerID="8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4" exitCode=0 Jul 28 15:49:28.429732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:28.429368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerDied","Data":"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4"} Jul 28 15:49:29.437302 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:29.437268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerStarted","Data":"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973"} Jul 28 15:49:29.437676 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:29.437334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerStarted","Data":"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b"} Jul 28 15:49:29.437676 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:29.437362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerStarted","Data":"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153"} Jul 28 15:49:29.437676 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:29.437563 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:29.461460 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:29.461411 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podStartSLOduration=2.553580461 podStartE2EDuration="6.461398789s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.907818328s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:49:29.460033407 +0000 UTC m=+956.141090420" watchObservedRunningTime="2026-07-28 15:49:29.461398789 +0000 UTC m=+956.142455800" Jul 28 15:49:30.445735 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:30.445696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:30.446117 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:30.445780 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:30.447341 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:30.447296 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:49:30.448120 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:30.448096 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:49:31.450890 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:31.450847 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:49:31.451318 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:31.451296 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:49:36.454547 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:36.454520 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:49:36.455136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:36.455103 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:49:36.455485 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:36.455464 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:49:46.455100 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:46.455058 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:49:46.455506 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:46.455475 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:49:56.455514 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:56.455415 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:49:56.455934 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:49:56.455911 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:06.455062 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:06.455014 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:06.455581 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:06.455475 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:16.455246 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:16.455203 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:16.455740 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:16.455686 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:26.455761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:26.455707 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:26.456189 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:26.456116 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:36.455672 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:36.455626 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:36.456134 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:36.456077 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:38.853491 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:38.853450 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:38.854077 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:38.853782 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:48.854866 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:48.854833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:50:48.855265 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:48.854937 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:50:58.402012 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.401981 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:50:58.402521 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.402219 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" containerID="cri-o://9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153" gracePeriod=30 Jul 28 15:50:58.402521 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.402310 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" containerID="cri-o://7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973" gracePeriod=30 Jul 28 15:50:58.402521 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.402343 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" containerID="cri-o://8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b" gracePeriod=30 Jul 28 15:50:58.475686 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.475653 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:50:58.479674 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.479655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.482198 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.482172 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4c735-predictor-serving-cert\"" Jul 28 15:50:58.482322 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.482267 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4c735-kube-rbac-proxy-sar-config\"" Jul 28 15:50:58.489385 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.489359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:50:58.591295 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.591262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.591483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.591308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.591483 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.591386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.591578 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.591474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.591578 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.591522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc6f\" (UniqueName: \"kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.692678 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.692630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.692678 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.692687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc6f\" (UniqueName: \"kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.692984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.692718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.692984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.692742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.692984 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.692768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.693220 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.693201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.693473 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.693452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.693515 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.693452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.695207 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.695185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.702526 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.702496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc6f\" (UniqueName: \"kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f\") pod \"isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.790745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.790711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:50:58.851671 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.851638 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5359347-8547-426c-a58a-7d8845de1e49" containerID="8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b" exitCode=2 Jul 28 15:50:58.851843 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.851695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerDied","Data":"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b"} Jul 28 15:50:58.854081 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.854013 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:50:58.854471 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.854445 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:50:58.929369 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:58.929238 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:50:58.931464 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:50:58.931441 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8fd62a_188c_4232_b04f_888e907a5b31.slice/crio-6c7e6a8eaee5bdddd06d5d70664ceb0ba5b64a809d07a17202b96840ccb47090 WatchSource:0}: Error finding container 6c7e6a8eaee5bdddd06d5d70664ceb0ba5b64a809d07a17202b96840ccb47090: Status 404 returned error can't find the container with id 6c7e6a8eaee5bdddd06d5d70664ceb0ba5b64a809d07a17202b96840ccb47090 Jul 28 15:50:59.859026 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:59.858983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerStarted","Data":"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029"} Jul 28 15:50:59.859469 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:50:59.859050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerStarted","Data":"6c7e6a8eaee5bdddd06d5d70664ceb0ba5b64a809d07a17202b96840ccb47090"} Jul 28 15:51:01.451109 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:01.451073 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:02.873268 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:02.873179 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerID="664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029" exitCode=0 Jul 28 15:51:02.873268 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:02.873253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerDied","Data":"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029"} Jul 28 15:51:03.879384 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.879333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerStarted","Data":"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d"} Jul 28 15:51:03.879857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.879409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerStarted","Data":"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3"} Jul 28 15:51:03.879857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.879697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:51:03.879857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.879752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:51:03.881220 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.881193 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:03.881846 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.881815 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5359347-8547-426c-a58a-7d8845de1e49" containerID="9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153" exitCode=0 Jul 28 15:51:03.881922 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.881863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerDied","Data":"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153"} Jul 28 15:51:03.900869 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:03.900823 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podStartSLOduration=2.352541637 podStartE2EDuration="5.900810034s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.548268397s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:51:03.89847769 +0000 UTC m=+1050.579534699" watchObservedRunningTime="2026-07-28 15:51:03.900810034 +0000 UTC m=+1050.581867042" Jul 28 15:51:04.886621 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:04.886587 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:06.451520 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:06.451484 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:08.854012 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:08.853960 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:51:08.854406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:08.854263 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:51:09.891027 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:09.890997 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:51:09.891575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:09.891548 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:11.451263 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:11.451215 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:11.451782 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:11.451405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:51:16.451047 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:16.451011 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:18.853760 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:18.853702 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Jul 28 15:51:18.854225 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:18.853922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:51:18.854225 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:18.854023 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:51:18.854225 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:18.854186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:51:19.892369 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:19.892331 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:21.451543 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:21.451496 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:26.451352 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:26.451262 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Jul 28 15:51:28.597142 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.597115 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:51:28.647006 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.646973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert\") pod \"b5359347-8547-426c-a58a-7d8845de1e49\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " Jul 28 15:51:28.647154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") pod \"b5359347-8547-426c-a58a-7d8845de1e49\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " Jul 28 15:51:28.647154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647055 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config\") pod \"b5359347-8547-426c-a58a-7d8845de1e49\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " Jul 28 15:51:28.647154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647106 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location\") pod \"b5359347-8547-426c-a58a-7d8845de1e49\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " Jul 28 15:51:28.647154 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647148 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rjjz\" (UniqueName: \"kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz\") pod \"b5359347-8547-426c-a58a-7d8845de1e49\" (UID: \"b5359347-8547-426c-a58a-7d8845de1e49\") " Jul 28 15:51:28.647377 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647334 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5359347-8547-426c-a58a-7d8845de1e49" (UID: "b5359347-8547-426c-a58a-7d8845de1e49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:51:28.647470 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5359347-8547-426c-a58a-7d8845de1e49" (UID: "b5359347-8547-426c-a58a-7d8845de1e49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:51:28.647535 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.647465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5359347-8547-426c-a58a-7d8845de1e49" (UID: "b5359347-8547-426c-a58a-7d8845de1e49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:51:28.649239 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.649216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5359347-8547-426c-a58a-7d8845de1e49" (UID: "b5359347-8547-426c-a58a-7d8845de1e49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:51:28.649395 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.649368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz" (OuterVolumeSpecName: "kube-api-access-4rjjz") pod "b5359347-8547-426c-a58a-7d8845de1e49" (UID: "b5359347-8547-426c-a58a-7d8845de1e49"). InnerVolumeSpecName "kube-api-access-4rjjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:51:28.748240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.748158 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:51:28.748240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.748186 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5359347-8547-426c-a58a-7d8845de1e49-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:51:28.748240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.748196 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5359347-8547-426c-a58a-7d8845de1e49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:51:28.748240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.748205 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5359347-8547-426c-a58a-7d8845de1e49-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:51:28.748240 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:28.748213 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjjz\" (UniqueName: \"kubernetes.io/projected/b5359347-8547-426c-a58a-7d8845de1e49-kube-api-access-4rjjz\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:51:29.003950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.003855 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5359347-8547-426c-a58a-7d8845de1e49" containerID="7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973" exitCode=137 Jul 28 15:51:29.003950 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.003938 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" Jul 28 15:51:29.004175 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.003934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerDied","Data":"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973"} Jul 28 15:51:29.004175 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.004091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r" event={"ID":"b5359347-8547-426c-a58a-7d8845de1e49","Type":"ContainerDied","Data":"9394b095e76673614ad21df596113c67e9c76c7470e5b05a90b17022a40fe486"} Jul 28 15:51:29.004175 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.004123 2580 scope.go:117] "RemoveContainer" containerID="7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973" Jul 28 15:51:29.013108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.013092 2580 scope.go:117] "RemoveContainer" containerID="8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b" Jul 28 15:51:29.020823 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.020787 2580 scope.go:117] "RemoveContainer" containerID="9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153" Jul 28 15:51:29.027854 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.027834 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:51:29.028744 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.028729 2580 scope.go:117] "RemoveContainer" containerID="8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4" Jul 28 15:51:29.033938 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.033917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r"] Jul 28 15:51:29.038662 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.038648 2580 scope.go:117] "RemoveContainer" containerID="7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973" Jul 28 15:51:29.038935 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:51:29.038913 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973\": container with ID starting with 7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973 not found: ID does not exist" containerID="7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973" Jul 28 15:51:29.039013 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.038945 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973"} err="failed to get container status \"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973\": rpc error: code = NotFound desc = could not find container \"7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973\": container with ID starting with 7c42091b590c6ef30440b7952b6b7e3971698664aa4cd673fea3d13e26833973 not found: ID does not exist" Jul 28 15:51:29.039013 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.038968 2580 scope.go:117] "RemoveContainer" containerID="8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b" Jul 28 15:51:29.039226 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:51:29.039210 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b\": container with ID starting with 8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b not found: ID does not exist" containerID="8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b" Jul 28 15:51:29.039265 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.039232 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b"} err="failed to get container status \"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b\": rpc error: code = NotFound desc = could not find container \"8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b\": container with ID starting with 8014ff78f4048059c21e8af221d087f06d572108cbd6b2456191debc49d2da3b not found: ID does not exist" Jul 28 15:51:29.039265 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.039248 2580 scope.go:117] "RemoveContainer" containerID="9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153" Jul 28 15:51:29.039482 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:51:29.039467 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153\": container with ID starting with 9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153 not found: ID does not exist" containerID="9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153" Jul 28 15:51:29.039528 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.039485 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153"} err="failed to get container status \"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153\": rpc error: code = NotFound desc = could not find container \"9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153\": container with ID starting with 9ac55fa40fc234edb1bc8970206db19ddfd226682268d8efc8958e739f21c153 not found: ID does not exist" Jul 28 15:51:29.039528 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.039498 2580 scope.go:117] "RemoveContainer" containerID="8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4" Jul 28 15:51:29.039755 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:51:29.039737 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4\": container with ID starting with 8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4 not found: ID does not exist" containerID="8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4" Jul 28 15:51:29.039837 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.039763 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4"} err="failed to get container status \"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4\": rpc error: code = NotFound desc = could not find container \"8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4\": container with ID starting with 8c012d0851617b8f2bea5de8ec37ba735b353cd6e9f50e586c4e1d50755ba6d4 not found: ID does not exist" Jul 28 15:51:29.858897 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.858867 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5359347-8547-426c-a58a-7d8845de1e49" path="/var/lib/kubelet/pods/b5359347-8547-426c-a58a-7d8845de1e49/volumes" Jul 28 15:51:29.892365 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:29.892336 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:39.891575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:39.891534 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:49.891995 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:49.891960 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:51:59.891972 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:51:59.891934 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:09.892345 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:09.892301 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:15.858601 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:15.858560 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:25.859024 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:25.858981 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:35.858656 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:35.858610 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:45.859359 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:45.859323 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:52:55.858936 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:52:55.858851 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:53:05.858738 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:05.858699 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:53:15.858694 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:15.858655 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:53:24.853503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:24.853463 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:53:33.823265 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:33.823239 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:53:33.823719 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:33.823305 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:53:33.825914 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:33.825893 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:53:33.826025 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:33.825933 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:53:34.854931 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:34.854900 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:53:38.665233 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:38.665203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:53:38.665729 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:38.665428 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" containerID="cri-o://05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3" gracePeriod=30 Jul 28 15:53:38.665729 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:38.665465 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" containerID="cri-o://e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d" gracePeriod=30 Jul 28 15:53:39.616883 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:39.616849 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerID="e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d" exitCode=2 Jul 28 15:53:39.616883 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:39.616886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerDied","Data":"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d"} Jul 28 15:53:39.887440 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:39.887329 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused" Jul 28 15:53:43.848175 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848047 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:53:43.848785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848768 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848787 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848813 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="storage-initializer" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848819 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="storage-initializer" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848826 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848831 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848841 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" Jul 28 15:53:43.848871 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848846 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" Jul 28 15:53:43.849073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848911 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kserve-container" Jul 28 15:53:43.849073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848921 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="agent" Jul 28 15:53:43.849073 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.848929 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5359347-8547-426c-a58a-7d8845de1e49" containerName="kube-rbac-proxy" Jul 28 15:53:43.852348 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.852322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:43.855564 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.855533 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Jul 28 15:53:43.856080 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.856052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Jul 28 15:53:43.859151 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.859128 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:53:43.864491 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.864473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:53:43.864610 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.864597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.867173 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.867155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\"" Jul 28 15:53:43.873598 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.873573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:53:43.964675 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.964891 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:43.964891 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.964891 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:43.964891 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:43.965123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.965123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzw9\" (UniqueName: \"kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:43.965123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.964975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.965123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.965029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5jk\" (UniqueName: \"kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:43.965123 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:43.965053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.066404 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.066404 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzw9\" (UniqueName: \"kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5jk\" (UniqueName: \"kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.066640 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.067046 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.067046 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.066892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.067160 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.067143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.067395 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.067355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.067513 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.067471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.067569 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.067471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.067569 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.067526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.069199 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.069177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.069389 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.069371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.076231 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.076209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5jk\" (UniqueName: \"kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.076399 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.076380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzw9\" (UniqueName: \"kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9\") pod \"isvc-canary-create-predictor-cb85cf55c-skqgz\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.164007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.163979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:44.175915 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.175890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:44.313208 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.313179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:53:44.316161 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:53:44.316128 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51ba38a_1936_446e_bcd7_d8eef4c7fa55.slice/crio-455f4a308df97859eaf207870fc09349b9e753dbd5e7ea5072f59df615cc86e8 WatchSource:0}: Error finding container 455f4a308df97859eaf207870fc09349b9e753dbd5e7ea5072f59df615cc86e8: Status 404 returned error can't find the container with id 455f4a308df97859eaf207870fc09349b9e753dbd5e7ea5072f59df615cc86e8 Jul 28 15:53:44.341635 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.341608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:53:44.355903 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:53:44.355877 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d00e431_2942_405a_9149_1b3bb91bd756.slice/crio-c68bd9a2f3fe8b99118de289b46ac3fffbcb5d4a49b9f26fccaa3f16f449b507 WatchSource:0}: Error finding container c68bd9a2f3fe8b99118de289b46ac3fffbcb5d4a49b9f26fccaa3f16f449b507: Status 404 returned error can't find the container with id c68bd9a2f3fe8b99118de289b46ac3fffbcb5d4a49b9f26fccaa3f16f449b507 Jul 28 15:53:44.644300 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.644210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerStarted","Data":"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf"} Jul 28 15:53:44.644300 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.644282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerStarted","Data":"455f4a308df97859eaf207870fc09349b9e753dbd5e7ea5072f59df615cc86e8"} Jul 28 15:53:44.645883 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.645859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerStarted","Data":"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b"} Jul 28 15:53:44.646002 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.645915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerStarted","Data":"c68bd9a2f3fe8b99118de289b46ac3fffbcb5d4a49b9f26fccaa3f16f449b507"} Jul 28 15:53:44.853614 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.853577 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Jul 28 15:53:44.887788 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:44.887741 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused" Jul 28 15:53:48.420209 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.420184 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:53:48.506602 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.506571 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config\") pod \"2a8fd62a-188c-4232-b04f-888e907a5b31\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " Jul 28 15:53:48.506768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.506616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert\") pod \"2a8fd62a-188c-4232-b04f-888e907a5b31\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " Jul 28 15:53:48.506768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.506651 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjc6f\" (UniqueName: \"kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f\") pod \"2a8fd62a-188c-4232-b04f-888e907a5b31\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " Jul 28 15:53:48.506768 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.506756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location\") pod \"2a8fd62a-188c-4232-b04f-888e907a5b31\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " Jul 28 15:53:48.506910 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.506824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls\") pod \"2a8fd62a-188c-4232-b04f-888e907a5b31\" (UID: \"2a8fd62a-188c-4232-b04f-888e907a5b31\") " Jul 28 15:53:48.507026 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.507003 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a8fd62a-188c-4232-b04f-888e907a5b31" (UID: "2a8fd62a-188c-4232-b04f-888e907a5b31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:53:48.507026 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.507006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a8fd62a-188c-4232-b04f-888e907a5b31" (UID: "2a8fd62a-188c-4232-b04f-888e907a5b31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:53:48.507141 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.507034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a8fd62a-188c-4232-b04f-888e907a5b31" (UID: "2a8fd62a-188c-4232-b04f-888e907a5b31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:53:48.508927 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.508907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f" (OuterVolumeSpecName: "kube-api-access-gjc6f") pod "2a8fd62a-188c-4232-b04f-888e907a5b31" (UID: "2a8fd62a-188c-4232-b04f-888e907a5b31"). InnerVolumeSpecName "kube-api-access-gjc6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:53:48.509018 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.509000 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a8fd62a-188c-4232-b04f-888e907a5b31" (UID: "2a8fd62a-188c-4232-b04f-888e907a5b31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:53:48.608458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.608374 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:53:48.608458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.608400 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fd62a-188c-4232-b04f-888e907a5b31-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:53:48.608458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.608409 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjc6f\" (UniqueName: \"kubernetes.io/projected/2a8fd62a-188c-4232-b04f-888e907a5b31-kube-api-access-gjc6f\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:53:48.608458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.608417 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a8fd62a-188c-4232-b04f-888e907a5b31-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:53:48.608458 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.608427 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a8fd62a-188c-4232-b04f-888e907a5b31-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:53:48.667730 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.667698 2580 generic.go:356] "Generic (PLEG): container finished" podID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerID="c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf" exitCode=0 Jul 28 15:53:48.667923 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.667772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerDied","Data":"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf"} Jul 28 15:53:48.669689 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.669670 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerID="05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3" exitCode=0 Jul 28 15:53:48.669761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.669718 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" Jul 28 15:53:48.669761 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.669734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerDied","Data":"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3"} Jul 28 15:53:48.669872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.669774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69" event={"ID":"2a8fd62a-188c-4232-b04f-888e907a5b31","Type":"ContainerDied","Data":"6c7e6a8eaee5bdddd06d5d70664ceb0ba5b64a809d07a17202b96840ccb47090"} Jul 28 15:53:48.669872 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.669819 2580 scope.go:117] "RemoveContainer" containerID="e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d" Jul 28 15:53:48.671253 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.671236 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d00e431-2942-405a-9149-1b3bb91bd756" containerID="54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b" exitCode=0 Jul 28 15:53:48.671345 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.671297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerDied","Data":"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b"} Jul 28 15:53:48.678729 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.678705 2580 scope.go:117] "RemoveContainer" containerID="05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3" Jul 28 15:53:48.691856 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.691726 2580 scope.go:117] "RemoveContainer" containerID="664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029" Jul 28 15:53:48.701732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.701712 2580 scope.go:117] "RemoveContainer" containerID="e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d" Jul 28 15:53:48.702107 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:53:48.702084 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d\": container with ID starting with e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d not found: ID does not exist" containerID="e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d" Jul 28 15:53:48.702201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702120 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d"} err="failed to get container status \"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d\": rpc error: code = NotFound desc = could not find container \"e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d\": container with ID starting with e83bb08efca9fe175701a18e4f884fa062e229491e99230ba5a0bc8c020d371d not found: ID does not exist" Jul 28 15:53:48.702201 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702142 2580 scope.go:117] "RemoveContainer" containerID="05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3" Jul 28 15:53:48.702428 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:53:48.702411 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3\": container with ID starting with 05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3 not found: ID does not exist" containerID="05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3" Jul 28 15:53:48.702480 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702431 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3"} err="failed to get container status \"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3\": rpc error: code = NotFound desc = could not find container \"05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3\": container with ID starting with 05c0ee8499285959bc99fee04d1c30c0c3ae918e1dd0706b9d12aca829014dc3 not found: ID does not exist" Jul 28 15:53:48.702480 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702442 2580 scope.go:117] "RemoveContainer" containerID="664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029" Jul 28 15:53:48.702580 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702561 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:53:48.702680 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:53:48.702663 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029\": container with ID starting with 664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029 not found: ID does not exist" containerID="664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029" Jul 28 15:53:48.702731 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.702691 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029"} err="failed to get container status \"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029\": rpc error: code = NotFound desc = could not find container \"664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029\": container with ID starting with 664b8ab17eb126a5009fc80e9948c0efe6238b8c8a9d97039ee72ab7e0fc2029 not found: ID does not exist" Jul 28 15:53:48.707227 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:48.707204 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69"] Jul 28 15:53:49.679149 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.679117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerStarted","Data":"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445"} Jul 28 15:53:49.679610 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.679183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerStarted","Data":"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae"} Jul 28 15:53:49.679610 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.679520 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:49.681136 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.681116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerStarted","Data":"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4"} Jul 28 15:53:49.681238 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.681160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerStarted","Data":"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf"} Jul 28 15:53:49.681373 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.681355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:49.709503 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.709461 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podStartSLOduration=2.962337245 podStartE2EDuration="6.709447642s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747110397s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:53:49.708264772 +0000 UTC m=+1216.389321784" watchObservedRunningTime="2026-07-28 15:53:49.709447642 +0000 UTC m=+1216.390504711" Jul 28 15:53:49.729742 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.729703 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podStartSLOduration=3.096425006 podStartE2EDuration="6.729690684s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633265678s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:53:49.727947145 +0000 UTC m=+1216.409004158" watchObservedRunningTime="2026-07-28 15:53:49.729690684 +0000 UTC m=+1216.410747695" Jul 28 15:53:49.860702 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:49.860663 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" path="/var/lib/kubelet/pods/2a8fd62a-188c-4232-b04f-888e907a5b31/volumes" Jul 28 15:53:50.687241 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:50.687204 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:50.687707 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:50.687268 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:50.688415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:50.688384 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:53:50.688530 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:50.688433 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:53:51.693446 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:51.693404 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:53:51.693868 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:51.693528 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:53:56.697830 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:56.697777 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:53:56.698358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:56.697941 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:53:56.698358 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:56.698176 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:53:56.698484 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:53:56.698435 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:06.698697 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:06.698662 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:06.700905 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:06.698665 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:16.699092 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:16.699055 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:16.699530 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:16.699055 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:26.698726 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:26.698643 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:26.699161 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:26.698652 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:36.698212 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:36.698154 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:36.698691 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:36.698396 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:46.698365 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:46.698325 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:46.698785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:46.698328 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:56.698535 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:56.698500 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:54:56.698971 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:56.698513 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:54:57.853739 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:54:57.853687 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:55:06.699974 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:06.699944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:55:07.858474 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:07.858447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:55:13.953341 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.953303 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:55:13.954069 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.954004 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kube-rbac-proxy" containerID="cri-o://7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445" gracePeriod=30 Jul 28 15:55:13.954239 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.953921 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" containerID="cri-o://ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae" gracePeriod=30 Jul 28 15:55:13.956262 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.956238 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:55:13.956707 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.956661 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" containerID="cri-o://b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf" gracePeriod=30 Jul 28 15:55:13.957024 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:13.956742 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kube-rbac-proxy" containerID="cri-o://f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4" gracePeriod=30 Jul 28 15:55:14.091574 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:14.091544 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d00e431-2942-405a-9149-1b3bb91bd756" containerID="7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445" exitCode=2 Jul 28 15:55:14.091716 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:14.091627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerDied","Data":"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445"} Jul 28 15:55:14.093693 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:14.093671 2580 generic.go:356] "Generic (PLEG): container finished" podID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerID="f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4" exitCode=2 Jul 28 15:55:14.093827 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:14.093710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerDied","Data":"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4"} Jul 28 15:55:16.694497 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:16.694451 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused" Jul 28 15:55:16.694497 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:16.694487 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Jul 28 15:55:16.699216 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:16.699185 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Jul 28 15:55:17.858962 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:17.858909 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Jul 28 15:55:18.728108 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.728083 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:55:18.730813 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.730775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:55:18.876518 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876434 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert\") pod \"2d00e431-2942-405a-9149-1b3bb91bd756\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " Jul 28 15:55:18.876518 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876471 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls\") pod \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " Jul 28 15:55:18.876518 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config\") pod \"2d00e431-2942-405a-9149-1b3bb91bd756\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876558 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnzw9\" (UniqueName: \"kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9\") pod \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config\") pod \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876607 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls\") pod \"2d00e431-2942-405a-9149-1b3bb91bd756\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert\") pod \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.876994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d00e431-2942-405a-9149-1b3bb91bd756" (UID: "2d00e431-2942-405a-9149-1b3bb91bd756"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877005 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d00e431-2942-405a-9149-1b3bb91bd756" (UID: "2d00e431-2942-405a-9149-1b3bb91bd756"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877028 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c51ba38a-1936-446e-bcd7-d8eef4c7fa55" (UID: "c51ba38a-1936-446e-bcd7-d8eef4c7fa55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c51ba38a-1936-446e-bcd7-d8eef4c7fa55" (UID: "c51ba38a-1936-446e-bcd7-d8eef4c7fa55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5jk\" (UniqueName: \"kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk\") pod \"2d00e431-2942-405a-9149-1b3bb91bd756\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877097 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location\") pod \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\" (UID: \"c51ba38a-1936-446e-bcd7-d8eef4c7fa55\") " Jul 28 15:55:18.877158 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location\") pod \"2d00e431-2942-405a-9149-1b3bb91bd756\" (UID: \"2d00e431-2942-405a-9149-1b3bb91bd756\") " Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877480 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d00e431-2942-405a-9149-1b3bb91bd756" (UID: "2d00e431-2942-405a-9149-1b3bb91bd756"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877499 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d00e431-2942-405a-9149-1b3bb91bd756-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877514 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877527 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.877895 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.877569 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c51ba38a-1936-446e-bcd7-d8eef4c7fa55" (UID: "c51ba38a-1936-446e-bcd7-d8eef4c7fa55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:55:18.878991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.878956 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d00e431-2942-405a-9149-1b3bb91bd756" (UID: "2d00e431-2942-405a-9149-1b3bb91bd756"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:55:18.878991 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.878978 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9" (OuterVolumeSpecName: "kube-api-access-cnzw9") pod "c51ba38a-1936-446e-bcd7-d8eef4c7fa55" (UID: "c51ba38a-1936-446e-bcd7-d8eef4c7fa55"). InnerVolumeSpecName "kube-api-access-cnzw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:55:18.879128 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.878998 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c51ba38a-1936-446e-bcd7-d8eef4c7fa55" (UID: "c51ba38a-1936-446e-bcd7-d8eef4c7fa55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:55:18.879230 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.879210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk" (OuterVolumeSpecName: "kube-api-access-nj5jk") pod "2d00e431-2942-405a-9149-1b3bb91bd756" (UID: "2d00e431-2942-405a-9149-1b3bb91bd756"). InnerVolumeSpecName "kube-api-access-nj5jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978187 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5jk\" (UniqueName: \"kubernetes.io/projected/2d00e431-2942-405a-9149-1b3bb91bd756-kube-api-access-nj5jk\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978221 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978262 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d00e431-2942-405a-9149-1b3bb91bd756-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978277 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978292 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzw9\" (UniqueName: \"kubernetes.io/projected/c51ba38a-1936-446e-bcd7-d8eef4c7fa55-kube-api-access-cnzw9\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:18.978347 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:18.978305 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d00e431-2942-405a-9149-1b3bb91bd756-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:55:19.102763 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.102724 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:55:19.103150 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103131 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" Jul 28 15:55:19.103150 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103150 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103159 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103164 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103173 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103179 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103186 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103192 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103199 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103204 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103212 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103217 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103225 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103230 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103235 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103241 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="storage-initializer" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103250 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103257 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kube-rbac-proxy" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103324 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kserve-container" Jul 28 15:55:19.103321 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103333 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kube-rbac-proxy" Jul 28 15:55:19.103932 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8fd62a-188c-4232-b04f-888e907a5b31" containerName="kube-rbac-proxy" Jul 28 15:55:19.103932 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103351 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kserve-container" Jul 28 15:55:19.103932 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103357 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" containerName="kube-rbac-proxy" Jul 28 15:55:19.103932 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.103364 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerName="kserve-container" Jul 28 15:55:19.107565 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.107550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.110152 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.110133 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Jul 28 15:55:19.110273 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.110153 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Jul 28 15:55:19.116300 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.116264 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:55:19.120438 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.120415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:55:19.120575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.120541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.121506 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.121480 2580 generic.go:356] "Generic (PLEG): container finished" podID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" containerID="b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf" exitCode=0 Jul 28 15:55:19.121592 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.121526 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" Jul 28 15:55:19.121592 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.121561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerDied","Data":"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf"} Jul 28 15:55:19.121690 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.121614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz" event={"ID":"c51ba38a-1936-446e-bcd7-d8eef4c7fa55","Type":"ContainerDied","Data":"455f4a308df97859eaf207870fc09349b9e753dbd5e7ea5072f59df615cc86e8"} Jul 28 15:55:19.121690 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.121653 2580 scope.go:117] "RemoveContainer" containerID="f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4" Jul 28 15:55:19.123630 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.123414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Jul 28 15:55:19.124043 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.124016 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d00e431-2942-405a-9149-1b3bb91bd756" containerID="ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae" exitCode=0 Jul 28 15:55:19.124132 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.124118 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" Jul 28 15:55:19.124447 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.124421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerDied","Data":"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae"} Jul 28 15:55:19.124565 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.124472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7" event={"ID":"2d00e431-2942-405a-9149-1b3bb91bd756","Type":"ContainerDied","Data":"c68bd9a2f3fe8b99118de289b46ac3fffbcb5d4a49b9f26fccaa3f16f449b507"} Jul 28 15:55:19.131275 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.131252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:55:19.137536 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.137517 2580 scope.go:117] "RemoveContainer" containerID="b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf" Jul 28 15:55:19.147733 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.147709 2580 scope.go:117] "RemoveContainer" containerID="c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf" Jul 28 15:55:19.157271 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.157253 2580 scope.go:117] "RemoveContainer" containerID="f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4" Jul 28 15:55:19.157597 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.157579 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4\": container with ID starting with f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4 not found: ID does not exist" containerID="f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4" Jul 28 15:55:19.157668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.157610 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4"} err="failed to get container status \"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4\": rpc error: code = NotFound desc = could not find container \"f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4\": container with ID starting with f35e456afe15c677f2a197aad8dec6910e0a0d04acac63044ca0de33f70e6bb4 not found: ID does not exist" Jul 28 15:55:19.157668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.157635 2580 scope.go:117] "RemoveContainer" containerID="b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf" Jul 28 15:55:19.157938 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.157914 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf\": container with ID starting with b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf not found: ID does not exist" containerID="b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf" Jul 28 15:55:19.157988 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.157938 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf"} err="failed to get container status \"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf\": rpc error: code = NotFound desc = could not find container \"b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf\": container with ID starting with b5c75d1a5b89bf8e09e95579d72f82efe6cd97c69e19561ebece06d6ac901daf not found: ID does not exist" Jul 28 15:55:19.157988 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.157957 2580 scope.go:117] "RemoveContainer" containerID="c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf" Jul 28 15:55:19.158233 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.158208 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf\": container with ID starting with c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf not found: ID does not exist" containerID="c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf" Jul 28 15:55:19.158287 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.158242 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf"} err="failed to get container status \"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf\": rpc error: code = NotFound desc = could not find container \"c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf\": container with ID starting with c525179e9c64e4d0db8947b8bb39f122a051e04af7a3e1fd336e2ae991d7bddf not found: ID does not exist" Jul 28 15:55:19.158287 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.158266 2580 scope.go:117] "RemoveContainer" containerID="7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445" Jul 28 15:55:19.166229 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.166214 2580 scope.go:117] "RemoveContainer" containerID="ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae" Jul 28 15:55:19.173157 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.173134 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:55:19.174526 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.174506 2580 scope.go:117] "RemoveContainer" containerID="54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b" Jul 28 15:55:19.179332 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.179309 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz"] Jul 28 15:55:19.182962 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.182946 2580 scope.go:117] "RemoveContainer" containerID="7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445" Jul 28 15:55:19.183244 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.183225 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445\": container with ID starting with 7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445 not found: ID does not exist" containerID="7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445" Jul 28 15:55:19.183298 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.183256 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445"} err="failed to get container status \"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445\": rpc error: code = NotFound desc = could not find container \"7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445\": container with ID starting with 7d903043508c2d83e957f33e83b15ce062dde04dbca4319be9722c81e0a68445 not found: ID does not exist" Jul 28 15:55:19.183298 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.183273 2580 scope.go:117] "RemoveContainer" containerID="ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae" Jul 28 15:55:19.183529 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.183516 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae\": container with ID starting with ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae not found: ID does not exist" containerID="ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae" Jul 28 15:55:19.183575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.183531 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae"} err="failed to get container status \"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae\": rpc error: code = NotFound desc = could not find container \"ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae\": container with ID starting with ad3a52e6886fd30c45e486442a53768c3768aebe5f885de40ad9dccd4e0aefae not found: ID does not exist" Jul 28 15:55:19.183575 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.183543 2580 scope.go:117] "RemoveContainer" containerID="54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b" Jul 28 15:55:19.183778 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.183761 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b\": container with ID starting with 54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b not found: ID does not exist" containerID="54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b" Jul 28 15:55:19.183849 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.183783 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b"} err="failed to get container status \"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b\": rpc error: code = NotFound desc = could not find container \"54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b\": container with ID starting with 54d861fa4a24ae7f2ad9536a167cdc2a8f45fcc30f2afd8a0ff349476a81796b not found: ID does not exist" Jul 28 15:55:19.191006 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.190984 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:55:19.195406 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.195376 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7"] Jul 28 15:55:19.280480 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.280480 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn25\" (UniqueName: \"kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.280765 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.281040 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.281040 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.280880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc7pz\" (UniqueName: \"kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.381618 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.381618 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc7pz\" (UniqueName: \"kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.381885 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.381885 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn25\" (UniqueName: \"kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.381885 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.381885 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.382093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.382093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.382093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.382093 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.381994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.382093 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.382079 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found Jul 28 15:55:19.382343 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:55:19.382142 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls podName:56e76b1c-525a-4b6d-b7ef-92b643c83637 nodeName:}" failed. No retries permitted until 2026-07-28 15:55:19.882120448 +0000 UTC m=+1306.563177454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls") pod "isvc-canary-promote-v2-predictor-86766c4958-lh556" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found Jul 28 15:55:19.382343 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.382343 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.382522 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.382586 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.382679 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.382732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.382683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.384419 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.384400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.391726 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.391702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc7pz\" (UniqueName: \"kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz\") pod \"isvc-canary-promote-predictor-d9759ffd5-7n4r9\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.392134 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.392118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn25\" (UniqueName: \"kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.420032 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.420004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:19.555187 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.555163 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:55:19.556977 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:55:19.556950 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3a977e_ae01_4443_acf3_ab4598b733de.slice/crio-44dd45bfd7297c66374dbcce134d5b4c5a78396fbc26bc24478b52bc248b0457 WatchSource:0}: Error finding container 44dd45bfd7297c66374dbcce134d5b4c5a78396fbc26bc24478b52bc248b0457: Status 404 returned error can't find the container with id 44dd45bfd7297c66374dbcce134d5b4c5a78396fbc26bc24478b52bc248b0457 Jul 28 15:55:19.858016 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.857983 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d00e431-2942-405a-9149-1b3bb91bd756" path="/var/lib/kubelet/pods/2d00e431-2942-405a-9149-1b3bb91bd756/volumes" Jul 28 15:55:19.858486 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.858472 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51ba38a-1936-446e-bcd7-d8eef4c7fa55" path="/var/lib/kubelet/pods/c51ba38a-1936-446e-bcd7-d8eef4c7fa55/volumes" Jul 28 15:55:19.887940 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.887906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:19.890442 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:19.890420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-lh556\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:20.037514 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:20.037469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:20.134732 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:20.134688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerStarted","Data":"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf"} Jul 28 15:55:20.134892 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:20.134763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerStarted","Data":"44dd45bfd7297c66374dbcce134d5b4c5a78396fbc26bc24478b52bc248b0457"} Jul 28 15:55:20.183681 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:20.183651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:55:20.184361 ip-10-0-137-14 kubenswrapper[2580]: W0728 15:55:20.184336 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e76b1c_525a_4b6d_b7ef_92b643c83637.slice/crio-cd48105b2e0c9289edc4bc947559bb5659dff6e2ea9f2484590c838aab9365b0 WatchSource:0}: Error finding container cd48105b2e0c9289edc4bc947559bb5659dff6e2ea9f2484590c838aab9365b0: Status 404 returned error can't find the container with id cd48105b2e0c9289edc4bc947559bb5659dff6e2ea9f2484590c838aab9365b0 Jul 28 15:55:21.144274 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:21.144233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerStarted","Data":"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9"} Jul 28 15:55:21.144745 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:21.144311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerStarted","Data":"cd48105b2e0c9289edc4bc947559bb5659dff6e2ea9f2484590c838aab9365b0"} Jul 28 15:55:24.161559 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:24.161532 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerID="3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf" exitCode=0 Jul 28 15:55:24.161874 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:24.161576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerDied","Data":"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf"} Jul 28 15:55:25.168840 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.168783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerStarted","Data":"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1"} Jul 28 15:55:25.169283 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.168875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerStarted","Data":"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730"} Jul 28 15:55:25.169283 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.169237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:25.169415 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.169395 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:25.170499 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.170476 2580 generic.go:356] "Generic (PLEG): container finished" podID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerID="259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9" exitCode=0 Jul 28 15:55:25.170624 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.170539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerDied","Data":"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9"} Jul 28 15:55:25.170857 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.170835 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:55:25.189194 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:25.189150 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podStartSLOduration=2.400053143 podStartE2EDuration="6.189134162s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789081019s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:25.187524531 +0000 UTC m=+1311.868581536" watchObservedRunningTime="2026-07-28 15:55:25.189134162 +0000 UTC m=+1311.870191172" Jul 28 15:55:26.177668 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.177627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerStarted","Data":"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249"} Jul 28 15:55:26.178144 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.177701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerStarted","Data":"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8"} Jul 28 15:55:26.178144 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.178059 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:55:26.178275 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.178239 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:26.178383 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.178362 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:26.179638 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.179617 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:55:26.198325 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:26.198277 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podStartSLOduration=3.288380622 podStartE2EDuration="7.198260802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90988018s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:55:26.196450351 +0000 UTC m=+1312.877507364" watchObservedRunningTime="2026-07-28 15:55:26.198260802 +0000 UTC m=+1312.879317813" Jul 28 15:55:27.183041 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:27.183001 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:55:28.188090 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:28.188051 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:55:31.182591 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:31.182566 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:55:31.183299 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:31.183271 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:55:33.193524 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:33.193497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:55:33.194038 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:33.194012 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:55:41.183716 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:41.183671 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:55:43.194860 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:43.194826 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:55:51.183345 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:51.183306 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:55:53.194481 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:55:53.194437 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:56:01.183551 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:01.183448 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:56:03.194178 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:03.194119 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:56:11.183309 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:11.183272 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:56:13.194395 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:13.194354 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:56:21.183106 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:21.183065 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:56:23.194223 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:23.194183 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:56:31.184165 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:31.184120 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:56:33.194568 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:33.194524 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:56:41.184514 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:41.184486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:56:43.195624 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:43.195596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:56:49.224584 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:49.224548 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:56:49.225143 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:49.224775 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" containerID="cri-o://fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730" gracePeriod=30 Jul 28 15:56:49.225143 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:49.224820 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kube-rbac-proxy" containerID="cri-o://f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1" gracePeriod=30 Jul 28 15:56:49.586329 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:49.586247 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerID="f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1" exitCode=2 Jul 28 15:56:49.586329 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:49.586309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerDied","Data":"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1"} Jul 28 15:56:51.179007 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:51.178962 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Jul 28 15:56:51.183322 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:51.183292 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Jul 28 15:56:53.882695 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.882662 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:56:53.916517 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert\") pod \"aa3a977e-ae01-4443-acf3-ab4598b733de\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " Jul 28 15:56:53.916517 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916531 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls\") pod \"aa3a977e-ae01-4443-acf3-ab4598b733de\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " Jul 28 15:56:53.916785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location\") pod \"aa3a977e-ae01-4443-acf3-ab4598b733de\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " Jul 28 15:56:53.916785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc7pz\" (UniqueName: \"kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz\") pod \"aa3a977e-ae01-4443-acf3-ab4598b733de\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " Jul 28 15:56:53.916785 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config\") pod \"aa3a977e-ae01-4443-acf3-ab4598b733de\" (UID: \"aa3a977e-ae01-4443-acf3-ab4598b733de\") " Jul 28 15:56:53.916968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916896 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa3a977e-ae01-4443-acf3-ab4598b733de" (UID: "aa3a977e-ae01-4443-acf3-ab4598b733de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:56:53.916968 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa3a977e-ae01-4443-acf3-ab4598b733de" (UID: "aa3a977e-ae01-4443-acf3-ab4598b733de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:56:53.917052 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916977 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:56:53.917052 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.916994 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa3a977e-ae01-4443-acf3-ab4598b733de-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:56:53.917166 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.917135 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa3a977e-ae01-4443-acf3-ab4598b733de" (UID: "aa3a977e-ae01-4443-acf3-ab4598b733de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:56:53.918845 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.918776 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa3a977e-ae01-4443-acf3-ab4598b733de" (UID: "aa3a977e-ae01-4443-acf3-ab4598b733de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:56:53.918957 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:53.918903 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz" (OuterVolumeSpecName: "kube-api-access-mc7pz") pod "aa3a977e-ae01-4443-acf3-ab4598b733de" (UID: "aa3a977e-ae01-4443-acf3-ab4598b733de"). InnerVolumeSpecName "kube-api-access-mc7pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:56:54.018362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.018322 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc7pz\" (UniqueName: \"kubernetes.io/projected/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-api-access-mc7pz\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:56:54.018362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.018354 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa3a977e-ae01-4443-acf3-ab4598b733de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:56:54.018362 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.018368 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa3a977e-ae01-4443-acf3-ab4598b733de-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:56:54.611891 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.611857 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerID="fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730" exitCode=0 Jul 28 15:56:54.612066 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.611920 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" Jul 28 15:56:54.612066 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.611939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerDied","Data":"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730"} Jul 28 15:56:54.612066 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.611999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9" event={"ID":"aa3a977e-ae01-4443-acf3-ab4598b733de","Type":"ContainerDied","Data":"44dd45bfd7297c66374dbcce134d5b4c5a78396fbc26bc24478b52bc248b0457"} Jul 28 15:56:54.612066 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.612025 2580 scope.go:117] "RemoveContainer" containerID="f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1" Jul 28 15:56:54.621352 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.621335 2580 scope.go:117] "RemoveContainer" containerID="fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730" Jul 28 15:56:54.629014 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.628996 2580 scope.go:117] "RemoveContainer" containerID="3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf" Jul 28 15:56:54.636835 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.636817 2580 scope.go:117] "RemoveContainer" containerID="f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1" Jul 28 15:56:54.637193 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:56:54.637165 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1\": container with ID starting with f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1 not found: ID does not exist" containerID="f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1" Jul 28 15:56:54.637309 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.637207 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1"} err="failed to get container status \"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1\": rpc error: code = NotFound desc = could not find container \"f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1\": container with ID starting with f0f60a80bdaeb658f241ac0f8ea823735b76629a9eaaa23fc10c75c7c3ccead1 not found: ID does not exist" Jul 28 15:56:54.637309 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.637227 2580 scope.go:117] "RemoveContainer" containerID="fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730" Jul 28 15:56:54.637674 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:56:54.637602 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730\": container with ID starting with fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730 not found: ID does not exist" containerID="fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730" Jul 28 15:56:54.637674 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.637664 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730"} err="failed to get container status \"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730\": rpc error: code = NotFound desc = could not find container \"fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730\": container with ID starting with fdd8a05f094df970dada1a3be38317f13eeaaef5f956b96c38c4878bf7213730 not found: ID does not exist" Jul 28 15:56:54.637900 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.637687 2580 scope.go:117] "RemoveContainer" containerID="3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf" Jul 28 15:56:54.638190 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:56:54.638167 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf\": container with ID starting with 3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf not found: ID does not exist" containerID="3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf" Jul 28 15:56:54.638275 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.638199 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf"} err="failed to get container status \"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf\": rpc error: code = NotFound desc = could not find container \"3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf\": container with ID starting with 3fd805b992e9f157d9e7f10d0d6750bbf520f3f73f744e56aabd06e436ae7ccf not found: ID does not exist" Jul 28 15:56:54.639985 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.639967 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:56:54.643725 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:54.643706 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9"] Jul 28 15:56:55.858767 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:56:55.858730 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" path="/var/lib/kubelet/pods/aa3a977e-ae01-4443-acf3-ab4598b733de/volumes" Jul 28 15:58:29.488944 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:29.488906 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:58:29.489456 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:29.489149 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" containerID="cri-o://e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8" gracePeriod=30 Jul 28 15:58:29.489456 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:29.489189 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kube-rbac-proxy" containerID="cri-o://bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249" gracePeriod=30 Jul 28 15:58:30.048812 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:30.048762 2580 generic.go:356] "Generic (PLEG): container finished" podID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerID="bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249" exitCode=2 Jul 28 15:58:30.048989 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:30.048836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerDied","Data":"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249"} Jul 28 15:58:33.189222 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.189175 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Jul 28 15:58:33.194931 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.194898 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Jul 28 15:58:33.853776 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.853734 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:58:33.855166 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.855137 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 15:58:33.857308 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.857275 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:58:33.858598 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:33.858574 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 15:58:34.243253 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.243222 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:58:34.342852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.342780 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location\") pod \"56e76b1c-525a-4b6d-b7ef-92b643c83637\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " Jul 28 15:58:34.342852 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.342855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nn25\" (UniqueName: \"kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25\") pod \"56e76b1c-525a-4b6d-b7ef-92b643c83637\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " Jul 28 15:58:34.343117 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.342883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert\") pod \"56e76b1c-525a-4b6d-b7ef-92b643c83637\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " Jul 28 15:58:34.343117 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.342917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config\") pod \"56e76b1c-525a-4b6d-b7ef-92b643c83637\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " Jul 28 15:58:34.343117 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.342951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") pod \"56e76b1c-525a-4b6d-b7ef-92b643c83637\" (UID: \"56e76b1c-525a-4b6d-b7ef-92b643c83637\") " Jul 28 15:58:34.343288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.343170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56e76b1c-525a-4b6d-b7ef-92b643c83637" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:58:34.343288 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.343257 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56e76b1c-525a-4b6d-b7ef-92b643c83637" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:58:34.343376 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.343300 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56e76b1c-525a-4b6d-b7ef-92b643c83637" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:58:34.345145 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.345124 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56e76b1c-525a-4b6d-b7ef-92b643c83637" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 15:58:34.345145 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.345133 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25" (OuterVolumeSpecName: "kube-api-access-9nn25") pod "56e76b1c-525a-4b6d-b7ef-92b643c83637" (UID: "56e76b1c-525a-4b6d-b7ef-92b643c83637"). InnerVolumeSpecName "kube-api-access-9nn25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:58:34.443763 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.443734 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:58:34.443763 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.443762 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56e76b1c-525a-4b6d-b7ef-92b643c83637-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:58:34.443969 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.443773 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56e76b1c-525a-4b6d-b7ef-92b643c83637-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:58:34.443969 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.443782 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nn25\" (UniqueName: \"kubernetes.io/projected/56e76b1c-525a-4b6d-b7ef-92b643c83637-kube-api-access-9nn25\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:58:34.443969 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:34.443791 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56e76b1c-525a-4b6d-b7ef-92b643c83637-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 15:58:35.074990 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.074947 2580 generic.go:356] "Generic (PLEG): container finished" podID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerID="e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8" exitCode=0 Jul 28 15:58:35.075296 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.075012 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" Jul 28 15:58:35.075296 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.075036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerDied","Data":"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8"} Jul 28 15:58:35.075296 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.075115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556" event={"ID":"56e76b1c-525a-4b6d-b7ef-92b643c83637","Type":"ContainerDied","Data":"cd48105b2e0c9289edc4bc947559bb5659dff6e2ea9f2484590c838aab9365b0"} Jul 28 15:58:35.075296 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.075154 2580 scope.go:117] "RemoveContainer" containerID="bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249" Jul 28 15:58:35.084433 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.084418 2580 scope.go:117] "RemoveContainer" containerID="e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8" Jul 28 15:58:35.092028 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.092009 2580 scope.go:117] "RemoveContainer" containerID="259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9" Jul 28 15:58:35.099557 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.099535 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:58:35.100271 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.100256 2580 scope.go:117] "RemoveContainer" containerID="bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249" Jul 28 15:58:35.100542 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:58:35.100527 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249\": container with ID starting with bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249 not found: ID does not exist" containerID="bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249" Jul 28 15:58:35.100588 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.100551 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249"} err="failed to get container status \"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249\": rpc error: code = NotFound desc = could not find container \"bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249\": container with ID starting with bfa72e5b1434eff1d21524219cda3761249d285079d1158de22c31e1a2a10249 not found: ID does not exist" Jul 28 15:58:35.100588 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.100567 2580 scope.go:117] "RemoveContainer" containerID="e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8" Jul 28 15:58:35.100843 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:58:35.100823 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8\": container with ID starting with e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8 not found: ID does not exist" containerID="e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8" Jul 28 15:58:35.100896 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.100850 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8"} err="failed to get container status \"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8\": rpc error: code = NotFound desc = could not find container \"e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8\": container with ID starting with e1ad717b94e40c5743dba0eac8a9241796e0c3ba42dfa0c31eb06dab028cf3f8 not found: ID does not exist" Jul 28 15:58:35.100896 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.100866 2580 scope.go:117] "RemoveContainer" containerID="259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9" Jul 28 15:58:35.101089 ip-10-0-137-14 kubenswrapper[2580]: E0728 15:58:35.101068 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9\": container with ID starting with 259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9 not found: ID does not exist" containerID="259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9" Jul 28 15:58:35.101135 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.101098 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9"} err="failed to get container status \"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9\": rpc error: code = NotFound desc = could not find container \"259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9\": container with ID starting with 259936692fe60ecba5bc25188f1a9143133e134f4e048295c91ac0493ddd1db9 not found: ID does not exist" Jul 28 15:58:35.107829 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.107787 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556"] Jul 28 15:58:35.857981 ip-10-0-137-14 kubenswrapper[2580]: I0728 15:58:35.857948 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" path="/var/lib/kubelet/pods/56e76b1c-525a-4b6d-b7ef-92b643c83637/volumes" Jul 28 16:00:04.828850 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.828811 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829166 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="storage-initializer" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829178 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="storage-initializer" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829186 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="storage-initializer" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829192 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="storage-initializer" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829216 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829240 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829246 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kube-rbac-proxy" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829251 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kube-rbac-proxy" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829264 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829276 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kube-rbac-proxy" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kube-rbac-proxy" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829337 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829347 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kube-rbac-proxy" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829354 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3a977e-ae01-4443-acf3-ab4598b733de" containerName="kserve-container" Jul 28 16:00:04.829360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.829360 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e76b1c-525a-4b6d-b7ef-92b643c83637" containerName="kube-rbac-proxy" Jul 28 16:00:04.832493 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.832474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:04.836040 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.836011 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 16:00:04.842020 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.841989 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 16:00:04.842145 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.842053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Jul 28 16:00:04.842208 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.842150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 16:00:04.843081 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.842515 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 16:00:04.843081 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.842529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\"" Jul 28 16:00:04.843081 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.842908 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Jul 28 16:00:04.847358 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.847337 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:00:04.910438 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.910396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prljj\" (UniqueName: \"kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:04.910619 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.910470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:04.910619 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.910537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:04.910619 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.910574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:04.910793 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:04.910642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011132 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011321 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prljj\" (UniqueName: \"kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011321 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011321 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011321 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011662 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011908 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.011908 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.011883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.013956 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.013930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.021697 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.021675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prljj\" (UniqueName: \"kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.147498 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.147414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:05.280434 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.280405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:00:05.282942 ip-10-0-137-14 kubenswrapper[2580]: W0728 16:00:05.282914 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610cc516_3945_463c_949a_7f6138e6d474.slice/crio-f91415a00fe0299e4e526eee179d6b7f6bc369c72d4b9755b50f89fe3c7195d5 WatchSource:0}: Error finding container f91415a00fe0299e4e526eee179d6b7f6bc369c72d4b9755b50f89fe3c7195d5: Status 404 returned error can't find the container with id f91415a00fe0299e4e526eee179d6b7f6bc369c72d4b9755b50f89fe3c7195d5 Jul 28 16:00:05.492677 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.492637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerStarted","Data":"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb"} Jul 28 16:00:05.492918 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:05.492701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerStarted","Data":"f91415a00fe0299e4e526eee179d6b7f6bc369c72d4b9755b50f89fe3c7195d5"} Jul 28 16:00:09.511621 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:09.511537 2580 generic.go:356] "Generic (PLEG): container finished" podID="610cc516-3945-463c-949a-7f6138e6d474" containerID="dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb" exitCode=0 Jul 28 16:00:09.511621 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:09.511611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerDied","Data":"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb"} Jul 28 16:00:10.524671 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:10.524630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerStarted","Data":"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882"} Jul 28 16:00:10.525113 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:10.524724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerStarted","Data":"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58"} Jul 28 16:00:10.525113 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:10.524939 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:10.546774 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:10.546727 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podStartSLOduration=2.640632992 podStartE2EDuration="6.546713182s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90608019s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:00:10.544022314 +0000 UTC m=+1597.225079327" watchObservedRunningTime="2026-07-28 16:00:10.546713182 +0000 UTC m=+1597.227770262" Jul 28 16:00:11.529288 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:11.529259 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:11.530947 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:11.530918 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:12.533116 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:12.533079 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:17.538222 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:17.538195 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:00:17.538723 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:17.538698 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:27.538688 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:27.538594 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:37.539361 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:37.539322 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:47.539626 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:47.539586 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:00:57.539590 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:00:57.539548 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:01:07.539210 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:07.539170 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:01:17.539591 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:17.539542 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:01:27.539976 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:27.539943 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:01:34.966946 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:34.966914 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:01:34.967434 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:34.967234 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" containerID="cri-o://697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58" gracePeriod=30 Jul 28 16:01:34.967434 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:34.967341 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kube-rbac-proxy" containerID="cri-o://7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882" gracePeriod=30 Jul 28 16:01:35.917903 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:35.917872 2580 generic.go:356] "Generic (PLEG): container finished" podID="610cc516-3945-463c-949a-7f6138e6d474" containerID="7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882" exitCode=2 Jul 28 16:01:35.918087 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:35.917951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerDied","Data":"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882"} Jul 28 16:01:37.534362 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:37.534314 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Jul 28 16:01:37.538827 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:37.538784 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Jul 28 16:01:39.617034 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.617011 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:01:39.635757 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.635734 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert\") pod \"610cc516-3945-463c-949a-7f6138e6d474\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " Jul 28 16:01:39.635925 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.635793 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config\") pod \"610cc516-3945-463c-949a-7f6138e6d474\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " Jul 28 16:01:39.635925 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.635831 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location\") pod \"610cc516-3945-463c-949a-7f6138e6d474\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " Jul 28 16:01:39.635925 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.635877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prljj\" (UniqueName: \"kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj\") pod \"610cc516-3945-463c-949a-7f6138e6d474\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " Jul 28 16:01:39.635925 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.635903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls\") pod \"610cc516-3945-463c-949a-7f6138e6d474\" (UID: \"610cc516-3945-463c-949a-7f6138e6d474\") " Jul 28 16:01:39.636199 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.636176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "610cc516-3945-463c-949a-7f6138e6d474" (UID: "610cc516-3945-463c-949a-7f6138e6d474"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:01:39.636261 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.636197 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "610cc516-3945-463c-949a-7f6138e6d474" (UID: "610cc516-3945-463c-949a-7f6138e6d474"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 16:01:39.636261 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.636201 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "610cc516-3945-463c-949a-7f6138e6d474" (UID: "610cc516-3945-463c-949a-7f6138e6d474"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 16:01:39.638176 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.638150 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "610cc516-3945-463c-949a-7f6138e6d474" (UID: "610cc516-3945-463c-949a-7f6138e6d474"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 16:01:39.638288 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.638227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj" (OuterVolumeSpecName: "kube-api-access-prljj") pod "610cc516-3945-463c-949a-7f6138e6d474" (UID: "610cc516-3945-463c-949a-7f6138e6d474"). InnerVolumeSpecName "kube-api-access-prljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 16:01:39.737238 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.737156 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/610cc516-3945-463c-949a-7f6138e6d474-proxy-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 16:01:39.737238 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.737185 2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-cabundle-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 16:01:39.737238 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.737196 2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/610cc516-3945-463c-949a-7f6138e6d474-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 16:01:39.737238 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.737204 2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610cc516-3945-463c-949a-7f6138e6d474-kserve-provision-location\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 16:01:39.737238 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.737213 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prljj\" (UniqueName: \"kubernetes.io/projected/610cc516-3945-463c-949a-7f6138e6d474-kube-api-access-prljj\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Jul 28 16:01:39.937301 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.937258 2580 generic.go:356] "Generic (PLEG): container finished" podID="610cc516-3945-463c-949a-7f6138e6d474" containerID="697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58" exitCode=0 Jul 28 16:01:39.937472 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.937313 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" Jul 28 16:01:39.937472 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.937351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerDied","Data":"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58"} Jul 28 16:01:39.937472 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.937404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb" event={"ID":"610cc516-3945-463c-949a-7f6138e6d474","Type":"ContainerDied","Data":"f91415a00fe0299e4e526eee179d6b7f6bc369c72d4b9755b50f89fe3c7195d5"} Jul 28 16:01:39.937472 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.937429 2580 scope.go:117] "RemoveContainer" containerID="7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882" Jul 28 16:01:39.949540 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.949518 2580 scope.go:117] "RemoveContainer" containerID="697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58" Jul 28 16:01:39.957728 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.957710 2580 scope.go:117] "RemoveContainer" containerID="dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb" Jul 28 16:01:39.958313 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.958293 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:01:39.961996 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.961974 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb"] Jul 28 16:01:39.966599 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.966583 2580 scope.go:117] "RemoveContainer" containerID="7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882" Jul 28 16:01:39.966901 ip-10-0-137-14 kubenswrapper[2580]: E0728 16:01:39.966883 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882\": container with ID starting with 7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882 not found: ID does not exist" containerID="7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882" Jul 28 16:01:39.966956 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.966910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882"} err="failed to get container status \"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882\": rpc error: code = NotFound desc = could not find container \"7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882\": container with ID starting with 7c241bb79074f93996d4c6eb7e9cbaca98543a776565644c4d0d08a40f81b882 not found: ID does not exist" Jul 28 16:01:39.966956 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.966928 2580 scope.go:117] "RemoveContainer" containerID="697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58" Jul 28 16:01:39.967193 ip-10-0-137-14 kubenswrapper[2580]: E0728 16:01:39.967173 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58\": container with ID starting with 697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58 not found: ID does not exist" containerID="697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58" Jul 28 16:01:39.967242 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.967199 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58"} err="failed to get container status \"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58\": rpc error: code = NotFound desc = could not find container \"697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58\": container with ID starting with 697b0580f4ed7e4f316d160ecbe583533e73822dc8d525cc266f123296b0ba58 not found: ID does not exist" Jul 28 16:01:39.967242 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.967222 2580 scope.go:117] "RemoveContainer" containerID="dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb" Jul 28 16:01:39.967430 ip-10-0-137-14 kubenswrapper[2580]: E0728 16:01:39.967415 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb\": container with ID starting with dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb not found: ID does not exist" containerID="dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb" Jul 28 16:01:39.967470 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:39.967436 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb"} err="failed to get container status \"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb\": rpc error: code = NotFound desc = could not find container \"dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb\": container with ID starting with dd8c700a9763202a41d7b975bca97827cc3fe9b9fc8e5af52672cff7529529eb not found: ID does not exist" Jul 28 16:01:41.858956 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:01:41.858922 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610cc516-3945-463c-949a-7f6138e6d474" path="/var/lib/kubelet/pods/610cc516-3945-463c-949a-7f6138e6d474/volumes" Jul 28 16:03:33.882229 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:03:33.882200 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 16:03:33.885113 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:03:33.885093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 16:03:33.885747 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:03:33.885728 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 16:03:33.888199 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:03:33.888184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 16:08:33.908628 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:08:33.908596 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 16:08:33.910944 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:08:33.910920 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 16:08:33.912843 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:08:33.912823 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 16:08:33.915216 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:08:33.915198 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 16:09:27.171087 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:27.171014 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cbcgv_29e3fd0b-5634-4368-a41a-2fa9768aa695/global-pull-secret-syncer/0.log" Jul 28 16:09:27.351207 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:27.351173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fvkrk_252fea0f-b42f-4b9f-be6a-552ca4ce7fe6/konnectivity-agent/0.log" Jul 28 16:09:27.464052 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:27.463968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-14.ec2.internal_372e08fc4324f245307306950cbd07f2/haproxy/0.log" Jul 28 16:09:30.943069 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:30.943042 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/alertmanager/0.log" Jul 28 16:09:30.978360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:30.978330 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/config-reloader/0.log" Jul 28 16:09:31.013317 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.013285 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/kube-rbac-proxy-web/0.log" Jul 28 16:09:31.053188 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.053162 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/kube-rbac-proxy/0.log" Jul 28 16:09:31.096267 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.096241 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/kube-rbac-proxy-metric/0.log" Jul 28 16:09:31.158331 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.158304 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/prom-label-proxy/0.log" Jul 28 16:09:31.192908 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.192882 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ae083da-da86-4cf5-a7ac-fec1dcf6704d/init-config-reloader/0.log" Jul 28 16:09:31.239699 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.239620 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-lwjpp_e8143238-a595-4467-81b2-0ee06280cd4d/cluster-monitoring-operator/0.log" Jul 28 16:09:31.292180 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.292148 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f89mg_64fee8a1-9410-4b99-80e1-acf4cabbafc5/kube-state-metrics/0.log" Jul 28 16:09:31.326261 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.326233 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f89mg_64fee8a1-9410-4b99-80e1-acf4cabbafc5/kube-rbac-proxy-main/0.log" Jul 28 16:09:31.357646 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.357621 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f89mg_64fee8a1-9410-4b99-80e1-acf4cabbafc5/kube-rbac-proxy-self/0.log" Jul 28 16:09:31.672259 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.672228 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8b2k_a31d5bef-2243-4d1e-8f68-a39f99cbe67f/node-exporter/0.log" Jul 28 16:09:31.708157 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.708129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8b2k_a31d5bef-2243-4d1e-8f68-a39f99cbe67f/kube-rbac-proxy/0.log" Jul 28 16:09:31.737012 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:31.736986 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8b2k_a31d5bef-2243-4d1e-8f68-a39f99cbe67f/init-textfile/0.log" Jul 28 16:09:34.266360 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:34.266291 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/1.log" Jul 28 16:09:34.270418 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:34.270394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-s6mqj_891844af-0fb5-4513-b06b-4c23d070cc2e/console-operator/2.log" Jul 28 16:09:35.289879 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.289854 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-h5h2b_2579dcd3-2e50-45eb-bef3-de71d89fcfb9/volume-data-source-validator/0.log" Jul 28 16:09:35.637638 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637560 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd"] Jul 28 16:09:35.637913 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637900 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kube-rbac-proxy" Jul 28 16:09:35.637968 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637914 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kube-rbac-proxy" Jul 28 16:09:35.637968 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637928 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="storage-initializer" Jul 28 16:09:35.637968 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637934 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="storage-initializer" Jul 28 16:09:35.637968 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" Jul 28 16:09:35.637968 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.637946 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" Jul 28 16:09:35.638120 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.638011 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kserve-container" Jul 28 16:09:35.638120 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.638019 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="610cc516-3945-463c-949a-7f6138e6d474" containerName="kube-rbac-proxy" Jul 28 16:09:35.640915 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.640896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.643696 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.643675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zm2w\"/\"kube-root-ca.crt\"" Jul 28 16:09:35.644963 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.644943 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zm2w\"/\"default-dockercfg-l8dtj\"" Jul 28 16:09:35.645062 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.644965 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zm2w\"/\"openshift-service-ca.crt\"" Jul 28 16:09:35.651473 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.651420 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd"] Jul 28 16:09:35.732751 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.732721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-proc\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.732957 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.732761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq825\" (UniqueName: \"kubernetes.io/projected/adae6343-f140-454e-bbe3-b832ffa564c9-kube-api-access-nq825\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.732957 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.732824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-sys\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.732957 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.732864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-lib-modules\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.732957 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.732879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-podres\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833312 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-sys\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833435 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-lib-modules\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833435 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-sys\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833505 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-lib-modules\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833505 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-podres\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833569 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-proc\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833569 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq825\" (UniqueName: \"kubernetes.io/projected/adae6343-f140-454e-bbe3-b832ffa564c9-kube-api-access-nq825\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833631 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-podres\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.833665 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.833634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/adae6343-f140-454e-bbe3-b832ffa564c9-proc\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.843247 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.843226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq825\" (UniqueName: \"kubernetes.io/projected/adae6343-f140-454e-bbe3-b832ffa564c9-kube-api-access-nq825\") pod \"perf-node-gather-daemonset-hqqmd\" (UID: \"adae6343-f140-454e-bbe3-b832ffa564c9\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:35.950609 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:35.950571 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:36.093145 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:36.093094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd"] Jul 28 16:09:36.095286 ip-10-0-137-14 kubenswrapper[2580]: W0728 16:09:36.095258 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podadae6343_f140_454e_bbe3_b832ffa564c9.slice/crio-3584226db1230f691c4de8f88e4a6ff905d6ebdd643eb9bc05c60b73b5bb6abc WatchSource:0}: Error finding container 3584226db1230f691c4de8f88e4a6ff905d6ebdd643eb9bc05c60b73b5bb6abc: Status 404 returned error can't find the container with id 3584226db1230f691c4de8f88e4a6ff905d6ebdd643eb9bc05c60b73b5bb6abc Jul 28 16:09:36.322287 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:36.322205 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5mtj5_92fdb4d9-0e11-4c05-99fc-39cb652c4d98/dns/0.log" Jul 28 16:09:36.359420 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:36.359380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5mtj5_92fdb4d9-0e11-4c05-99fc-39cb652c4d98/kube-rbac-proxy/0.log" Jul 28 16:09:36.515046 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:36.515010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xz94w_4e57260c-da09-4286-8a99-5739e4d5d6d5/dns-node-resolver/0.log" Jul 28 16:09:37.071642 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.071614 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85bv6_5968e5c7-27a6-4b49-b0af-d89ee40cdeab/node-ca/0.log" Jul 28 16:09:37.087047 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.087011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" event={"ID":"adae6343-f140-454e-bbe3-b832ffa564c9","Type":"ContainerStarted","Data":"4671b6fce67d2d6f95c33845965dd064dcdfd6c12e7dc6aa9cc3e4c5acecdef2"} Jul 28 16:09:37.087208 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.087058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" event={"ID":"adae6343-f140-454e-bbe3-b832ffa564c9","Type":"ContainerStarted","Data":"3584226db1230f691c4de8f88e4a6ff905d6ebdd643eb9bc05c60b73b5bb6abc"} Jul 28 16:09:37.087208 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.087116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:37.103948 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.103896 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" podStartSLOduration=2.103881011 podStartE2EDuration="2.103881011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:37.102536964 +0000 UTC m=+2163.783593977" watchObservedRunningTime="2026-07-28 16:09:37.103881011 +0000 UTC m=+2163.784938022" Jul 28 16:09:37.844409 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:37.844380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57cdcfc679-kw468_8975da34-3728-4e3d-8f6a-e782583c3e94/router/0.log" Jul 28 16:09:38.220890 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.220863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j5ftk_66dab572-5fdd-4a86-ba23-8b43f447b27a/serve-healthcheck-canary/0.log" Jul 28 16:09:38.561866 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.561753 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lgx6t_ad57ff0f-57a8-4509-9a41-329fa0b76aa4/insights-operator/0.log" Jul 28 16:09:38.562656 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.562642 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lgx6t_ad57ff0f-57a8-4509-9a41-329fa0b76aa4/insights-operator/1.log" Jul 28 16:09:38.652888 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.652862 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdn8k_c1889b55-d339-4731-a314-e58ea353cf0c/kube-rbac-proxy/0.log" Jul 28 16:09:38.725638 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.725611 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdn8k_c1889b55-d339-4731-a314-e58ea353cf0c/exporter/0.log" Jul 28 16:09:38.775276 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:38.775246 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kdn8k_c1889b55-d339-4731-a314-e58ea353cf0c/extractor/0.log" Jul 28 16:09:41.045041 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:41.045015 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85d8b8ddfc-jpk7t_cb267dde-5e85-483c-a2ed-b7b52a141b0c/manager/0.log" Jul 28 16:09:41.211821 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:41.211772 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-pdnrk_ec5fbd9f-706c-4c94-a3cd-cb9fa831b382/manager/0.log" Jul 28 16:09:41.242321 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:41.242292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7kls2_b9ee021a-e5d1-4a48-934e-eab4b4efe2eb/s3-init/0.log" Jul 28 16:09:41.259516 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:41.259490 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7kls2_b9ee021a-e5d1-4a48-934e-eab4b4efe2eb/download-hf-model/0.log" Jul 28 16:09:41.288776 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:41.288754 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-xft2l_03971e83-2e78-4725-ab35-e345215cc1e2/seaweedfs/0.log" Jul 28 16:09:43.104482 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:43.104449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-hqqmd" Jul 28 16:09:45.752284 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:45.752252 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9_398680a4-2b5a-4ee2-9a9f-3ec51e80043c/kube-storage-version-migrator-operator/1.log" Jul 28 16:09:45.753274 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:45.753248 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-4qfb9_398680a4-2b5a-4ee2-9a9f-3ec51e80043c/kube-storage-version-migrator-operator/0.log" Jul 28 16:09:46.843198 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.843172 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/kube-multus-additional-cni-plugins/0.log" Jul 28 16:09:46.871474 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.871373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/egress-router-binary-copy/0.log" Jul 28 16:09:46.898773 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.898752 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/cni-plugins/0.log" Jul 28 16:09:46.929387 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.929365 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/bond-cni-plugin/0.log" Jul 28 16:09:46.952200 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.952176 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/routeoverride-cni/0.log" Jul 28 16:09:46.974440 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.974412 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/whereabouts-cni-bincopy/0.log" Jul 28 16:09:46.999372 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:46.999348 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbs4j_4ee68990-196b-47a9-bf0a-397cc21718da/whereabouts-cni/0.log" Jul 28 16:09:47.294184 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:47.294153 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xpdcx_5cc89da4-ef78-4830-9c8d-6e53ff954440/kube-multus/0.log" Jul 28 16:09:47.410485 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:47.410457 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gs4v8_79c18d1a-bf4d-4ced-987a-78745c891505/network-metrics-daemon/0.log" Jul 28 16:09:47.430414 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:47.430390 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gs4v8_79c18d1a-bf4d-4ced-987a-78745c891505/kube-rbac-proxy/0.log" Jul 28 16:09:48.378987 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.378962 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-controller/0.log" Jul 28 16:09:48.398172 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.398150 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/1.log" Jul 28 16:09:48.398172 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.398164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovn-acl-logging/0.log" Jul 28 16:09:48.417105 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.417083 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/kube-rbac-proxy-node/0.log" Jul 28 16:09:48.442726 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.442699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 16:09:48.466126 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.466100 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/northd/0.log" Jul 28 16:09:48.484034 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.484011 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/nbdb/0.log" Jul 28 16:09:48.508839 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.508814 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/sbdb/0.log" Jul 28 16:09:48.557209 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:48.557187 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hv4d_e25eba0e-6332-4616-84e8-24ad2a645237/ovnkube-controller/0.log" Jul 28 16:09:49.764767 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:49.764742 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-rtpzf_832386bd-d9d4-477c-a341-d7187c9a08e5/check-endpoints/0.log" Jul 28 16:09:49.841355 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:49.841331 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hxplb_727244f6-be9a-4e62-8294-29ef4c4d7936/network-check-target-container/0.log" Jul 28 16:09:50.762524 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:50.762494 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rthgn_c613a05c-0bc3-4bfa-ac92-7ff568727e95/iptables-alerter/0.log" Jul 28 16:09:51.517000 ip-10-0-137-14 kubenswrapper[2580]: I0728 16:09:51.516973 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-qmn8h_08026358-1001-4b5c-ad0c-7da3c7128309/tuned/0.log"