May 20 13:47:55.245167 ip-10-0-137-148 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory May 20 13:47:55.245176 ip-10-0-137-148 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory May 20 13:47:55.245184 ip-10-0-137-148 systemd[1]: kubelet.service: Failed with result 'resources'. May 20 13:47:55.245434 ip-10-0-137-148 systemd[1]: Failed to start Kubernetes Kubelet. May 20 13:48:05.035078 ip-10-0-137-148 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. May 20 13:48:05.035094 ip-10-0-137-148 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fbe5d13b364a4e98a3bfcaf921fcf84f -- May 20 13:50:15.122069 ip-10-0-137-148 systemd[1]: Starting Kubernetes Kubelet... May 20 13:50:15.601725 ip-10-0-137-148 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:15.601725 ip-10-0-137-148 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 20 13:50:15.601725 ip-10-0-137-148 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:15.601725 ip-10-0-137-148 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 20 13:50:15.601725 ip-10-0-137-148 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:15.603962 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.603803 2575 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 20 13:50:15.609635 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609616 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:15.609635 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609635 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609641 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609644 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609647 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609650 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609652 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609655 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609658 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609661 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609664 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609667 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609669 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609672 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609675 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609677 2575 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609680 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609682 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609685 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609690 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609692 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:15.609709 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609695 2575 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609697 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609700 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609702 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609705 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609708 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609710 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609713 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609716 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609719 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609722 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609725 2575 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609728 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609730 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609733 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609735 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609738 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609754 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609758 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:15.610196 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609761 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609764 2575 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609766 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609769 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609771 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609774 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609776 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609779 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609781 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609783 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609786 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609788 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609791 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609793 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609798 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609804 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609807 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609811 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609814 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609817 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:15.610679 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609819 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609822 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609824 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609827 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609829 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609832 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609834 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609837 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609839 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609842 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609845 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609848 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609850 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609852 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609855 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609857 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609860 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609863 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609866 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609868 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:15.611177 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609872 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609875 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609877 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609880 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609883 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.609886 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610307 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610313 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610317 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610321 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610324 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610326 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610329 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610332 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610334 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610337 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610339 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610342 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610345 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:15.611656 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610348 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610350 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610353 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610356 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610359 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610362 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610364 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610367 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610370 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610372 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610375 2575 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610377 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610380 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610382 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610385 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610387 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610391 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610393 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610396 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610399 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:15.612133 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610402 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610404 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610407 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610409 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610411 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610414 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610417 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610419 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610423 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610426 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610429 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610431 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610434 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610436 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610439 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610441 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610444 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610447 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610449 2575 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610452 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:15.612634 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610454 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610457 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610460 2575 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610462 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610465 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610467 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610470 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610472 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610475 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610479 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610483 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610487 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610489 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610492 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610494 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610497 2575 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610500 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610502 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610505 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:15.613136 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610507 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610511 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610513 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610516 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610518 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610520 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610523 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610526 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610528 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610530 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610534 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610536 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610539 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.610541 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610622 2575 flags.go:64] FLAG: --address="0.0.0.0" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610637 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610644 2575 flags.go:64] FLAG: --anonymous-auth="true" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610649 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610654 2575 flags.go:64] FLAG: --authentication-token-webhook="false" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610657 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610662 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 20 13:50:15.613604 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610666 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610670 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610673 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610677 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610681 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610684 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610688 2575 flags.go:64] FLAG: --cgroup-root="" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610691 2575 flags.go:64] FLAG: --cgroups-per-qos="true" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610694 2575 flags.go:64] FLAG: --client-ca-file="" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610698 2575 flags.go:64] FLAG: --cloud-config="" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610701 2575 flags.go:64] FLAG: --cloud-provider="external" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610704 2575 flags.go:64] FLAG: --cluster-dns="[]" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610708 2575 flags.go:64] FLAG: --cluster-domain="" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610711 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610714 2575 flags.go:64] FLAG: --config-dir="" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610717 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610721 2575 flags.go:64] FLAG: --container-log-max-files="5" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610725 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610728 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610731 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610735 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610738 2575 flags.go:64] FLAG: --contention-profiling="false" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610754 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610757 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610761 2575 flags.go:64] FLAG: --cpu-manager-policy="none" May 20 13:50:15.614125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610764 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610768 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610771 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610775 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610777 2575 flags.go:64] FLAG: --enable-load-reader="false" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610781 2575 flags.go:64] FLAG: --enable-server="true" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610784 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610791 2575 flags.go:64] FLAG: --event-burst="100" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610794 2575 flags.go:64] FLAG: --event-qps="50" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610797 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610801 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610804 2575 flags.go:64] FLAG: --eviction-hard="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610808 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610811 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610814 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610817 2575 flags.go:64] FLAG: --eviction-soft="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610820 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610823 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610826 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610829 2575 flags.go:64] FLAG: --experimental-mounter-path="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610832 2575 flags.go:64] FLAG: --fail-cgroupv1="false" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610835 2575 flags.go:64] FLAG: --fail-swap-on="true" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610837 2575 flags.go:64] FLAG: --feature-gates="" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610841 2575 flags.go:64] FLAG: --file-check-frequency="20s" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610844 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 20 13:50:15.614765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610848 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610851 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610854 2575 flags.go:64] FLAG: --healthz-port="10248" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610857 2575 flags.go:64] FLAG: --help="false" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610860 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-137-148.ec2.internal" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610864 2575 flags.go:64] FLAG: --housekeeping-interval="10s" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610868 2575 flags.go:64] FLAG: --http-check-frequency="20s" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610871 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610874 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610878 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610881 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610884 2575 flags.go:64] FLAG: --image-service-endpoint="" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610887 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610890 2575 flags.go:64] FLAG: --kube-api-burst="100" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610893 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610897 2575 flags.go:64] FLAG: --kube-api-qps="50" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610900 2575 flags.go:64] FLAG: --kube-reserved="" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610903 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610906 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610909 2575 flags.go:64] FLAG: --kubelet-cgroups="" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610912 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610915 2575 flags.go:64] FLAG: --lock-file="" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610918 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610921 2575 flags.go:64] FLAG: --log-flush-frequency="5s" May 20 13:50:15.615366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610924 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610930 2575 flags.go:64] FLAG: --log-json-split-stream="false" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610933 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610936 2575 flags.go:64] FLAG: --log-text-split-stream="false" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610939 2575 flags.go:64] FLAG: --logging-format="text" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610942 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610945 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610948 2575 flags.go:64] FLAG: --manifest-url="" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610951 2575 flags.go:64] FLAG: --manifest-url-header="" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610955 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610959 2575 flags.go:64] FLAG: --max-open-files="1000000" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610963 2575 flags.go:64] FLAG: --max-pods="110" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610966 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610970 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610973 2575 flags.go:64] FLAG: --memory-manager-policy="None" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610976 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610979 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610982 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610989 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.610997 2575 flags.go:64] FLAG: --node-status-max-images="50" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611001 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611004 2575 flags.go:64] FLAG: --oom-score-adj="-999" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611007 2575 flags.go:64] FLAG: --pod-cidr="" May 20 13:50:15.616114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611011 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611017 2575 flags.go:64] FLAG: --pod-manifest-path="" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611020 2575 flags.go:64] FLAG: --pod-max-pids="-1" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611023 2575 flags.go:64] FLAG: --pods-per-core="0" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611026 2575 flags.go:64] FLAG: --port="10250" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611029 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611032 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ce98fef9145209e" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611035 2575 flags.go:64] FLAG: --qos-reserved="" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611039 2575 flags.go:64] FLAG: --read-only-port="10255" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611042 2575 flags.go:64] FLAG: --register-node="true" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611045 2575 flags.go:64] FLAG: --register-schedulable="true" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611048 2575 flags.go:64] FLAG: --register-with-taints="" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611052 2575 flags.go:64] FLAG: --registry-burst="10" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611055 2575 flags.go:64] FLAG: --registry-qps="5" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611058 2575 flags.go:64] FLAG: --reserved-cpus="" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611061 2575 flags.go:64] FLAG: --reserved-memory="" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611065 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611068 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611071 2575 flags.go:64] FLAG: --rotate-certificates="false" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611074 2575 flags.go:64] FLAG: --rotate-server-certificates="false" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611077 2575 flags.go:64] FLAG: --runonce="false" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611080 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611083 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611086 2575 flags.go:64] FLAG: --seccomp-default="false" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611090 2575 flags.go:64] FLAG: --serialize-image-pulls="true" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611093 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 20 13:50:15.616671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611096 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611099 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611105 2575 flags.go:64] FLAG: --storage-driver-password="root" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611108 2575 flags.go:64] FLAG: --storage-driver-secure="false" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611111 2575 flags.go:64] FLAG: --storage-driver-table="stats" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611114 2575 flags.go:64] FLAG: --storage-driver-user="root" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611117 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611121 2575 flags.go:64] FLAG: --sync-frequency="1m0s" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611124 2575 flags.go:64] FLAG: --system-cgroups="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611127 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611132 2575 flags.go:64] FLAG: --system-reserved-cgroup="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611135 2575 flags.go:64] FLAG: --tls-cert-file="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611138 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611142 2575 flags.go:64] FLAG: --tls-min-version="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611145 2575 flags.go:64] FLAG: --tls-private-key-file="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611148 2575 flags.go:64] FLAG: --topology-manager-policy="none" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611151 2575 flags.go:64] FLAG: --topology-manager-policy-options="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611154 2575 flags.go:64] FLAG: --topology-manager-scope="container" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611157 2575 flags.go:64] FLAG: --v="2" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611162 2575 flags.go:64] FLAG: --version="false" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611166 2575 flags.go:64] FLAG: --vmodule="" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611171 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.611174 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611303 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611307 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:15.617333 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611311 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611314 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611316 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611319 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611322 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611325 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611328 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611330 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611333 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611337 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611340 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611342 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611345 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611348 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611351 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611355 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611358 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611361 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611364 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:15.617957 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611366 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611369 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611371 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611374 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611376 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611379 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611382 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611384 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611386 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611389 2575 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611392 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611394 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611397 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611400 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611402 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611405 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611407 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611410 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611412 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611415 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:15.618437 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611418 2575 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611421 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611425 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611428 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611431 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611433 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611436 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611439 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611441 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611444 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611446 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611449 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611451 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611454 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611456 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611459 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611461 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611464 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611466 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611469 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:15.618949 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611472 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611474 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611477 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611480 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611482 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611485 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611487 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611490 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611492 2575 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611495 2575 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611498 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611500 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611503 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611506 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611510 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611512 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611515 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611517 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611522 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611524 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:15.619445 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611527 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611529 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611533 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611537 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.611540 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.612295 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.619839 2575 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.619860 2575 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619939 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619945 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619948 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619952 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619955 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619958 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619961 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619964 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:15.619973 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619967 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619970 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619972 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619975 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619978 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619981 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619985 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619988 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619991 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619993 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619996 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.619998 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620001 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620004 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620006 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620009 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620011 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620013 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620016 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:15.620367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620018 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620021 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620023 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620026 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620030 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620033 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620036 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620038 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620041 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620043 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620046 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620049 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620053 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620056 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620059 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620062 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620065 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620067 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620070 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:15.620959 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620073 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620075 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620078 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620081 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620083 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620086 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620088 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620091 2575 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620093 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620096 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620099 2575 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620101 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620104 2575 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620106 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620109 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620112 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620114 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620117 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620123 2575 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620125 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:15.621427 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620128 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620130 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620133 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620135 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620139 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620141 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620144 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620146 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620149 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620152 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620155 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620158 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620162 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620167 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620170 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620173 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620176 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620179 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620181 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:15.621945 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620184 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.620189 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620336 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620342 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620345 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620348 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620350 2575 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620353 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620356 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620359 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620361 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620364 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620373 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620376 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620378 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:15.622426 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620387 2575 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620390 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620392 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620395 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620397 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620400 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620403 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620406 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620409 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620411 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620414 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620416 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620419 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620422 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620425 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620427 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620430 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620432 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620435 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620437 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:15.622811 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620440 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620442 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620445 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620447 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620450 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620452 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620455 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620457 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620460 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620462 2575 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620470 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620473 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620476 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620478 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620481 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620483 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620486 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620488 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620491 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620494 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:15.623295 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620497 2575 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620500 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620502 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620504 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620508 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620510 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620513 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620515 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620518 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620520 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620523 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620526 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620530 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620534 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620538 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620542 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620546 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620548 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620551 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:15.623801 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620554 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620557 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620559 2575 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620562 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620571 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620574 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620577 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620579 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620582 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620584 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620587 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620589 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620593 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:15.620595 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.620601 2575 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:15.624356 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.621482 2575 server.go:962] "Client rotation is on, will bootstrap in background" May 20 13:50:15.624871 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.624004 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 20 13:50:15.625051 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.625038 2575 server.go:1019] "Starting client certificate rotation" May 20 13:50:15.625160 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.625142 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 20 13:50:15.625193 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.625186 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 20 13:50:15.655469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.655444 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 20 13:50:15.660718 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.660690 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 20 13:50:15.675594 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.675564 2575 log.go:25] "Validated CRI v1 runtime API" May 20 13:50:15.682197 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.682170 2575 log.go:25] "Validated CRI v1 image API" May 20 13:50:15.684766 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.684728 2575 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 20 13:50:15.689958 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.689928 2575 fs.go:135] Filesystem UUIDs: map[42ea60bd-8daf-4dca-90ca-dd63b51224b0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f6ea5db0-c15a-4c17-83dd-e7d3178fd295:/dev/nvme0n1p3] May 20 13:50:15.690025 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.689959 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 20 13:50:15.694753 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.694619 2575 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:15.694098555 +0000 UTC m=+0.444910944 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099232 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c645f80cdeee5082ca5caa3a53e56 SystemUUID:ec2c645f-80cd-eee5-082c-a5caa3a53e56 BootID:fbe5d13b-364a-4e98-a3bf-caf921fcf84f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:a9:9d:0a:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:a9:9d:0a:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:e5:0f:29:ad:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 20 13:50:15.696326 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.696312 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 20 13:50:15.696423 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.696410 2575 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 20 13:50:15.699459 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.699431 2575 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 20 13:50:15.699617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.699462 2575 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 20 13:50:15.699666 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.699631 2575 topology_manager.go:138] "Creating topology manager with none policy" May 20 13:50:15.699666 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.699640 2575 container_manager_linux.go:306] "Creating device plugin manager" May 20 13:50:15.699666 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.699649 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 20 13:50:15.700506 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.700491 2575 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 20 13:50:15.702068 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.702056 2575 state_mem.go:36] "Initialized new in-memory state store" May 20 13:50:15.702194 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.702185 2575 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 20 13:50:15.703271 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.703255 2575 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 20 13:50:15.704783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.704773 2575 kubelet.go:491] "Attempting to sync node with API server" May 20 13:50:15.704818 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.704788 2575 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 20 13:50:15.704818 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.704801 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 20 13:50:15.704818 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.704810 2575 kubelet.go:397] "Adding apiserver pod source" May 20 13:50:15.704909 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.704820 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 20 13:50:15.706323 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.706309 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 20 13:50:15.706365 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.706330 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 20 13:50:15.710492 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.710458 2575 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 20 13:50:15.714814 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.714795 2575 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 20 13:50:15.715703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715690 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715709 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715717 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715722 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715731 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715736 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715758 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 20 13:50:15.715765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715766 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 20 13:50:15.715968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715782 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 20 13:50:15.715968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715789 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 20 13:50:15.715968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.715799 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 20 13:50:15.716198 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.716189 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 20 13:50:15.717765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.717756 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 20 13:50:15.717765 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.717765 2575 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 20 13:50:15.721010 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.720997 2575 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 20 13:50:15.721058 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.721040 2575 server.go:1295] "Started kubelet" May 20 13:50:15.721179 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.721125 2575 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 20 13:50:15.721240 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.721204 2575 server_v1.go:47] "podresources" method="list" useActivePods=true May 20 13:50:15.721537 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.721510 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 20 13:50:15.721926 ip-10-0-137-148 systemd[1]: Started Kubernetes Kubelet. May 20 13:50:15.722616 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.722601 2575 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 20 13:50:15.724644 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.724629 2575 server.go:317] "Adding debug handlers to kubelet server" May 20 13:50:15.724901 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.724874 2575 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 20 13:50:15.724901 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.724888 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 20 13:50:15.724995 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.724891 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 20 13:50:15.730330 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.730303 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 20 13:50:15.730644 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.730619 2575 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 20 13:50:15.730878 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.730865 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 20 13:50:15.730989 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.730971 2575 volume_manager.go:295] "The desired_state_of_world populator starts" May 20 13:50:15.731035 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.730994 2575 volume_manager.go:297] "Starting Kubelet Volume Manager" May 20 13:50:15.731195 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.731180 2575 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 20 13:50:15.731276 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.731264 2575 reconstruct.go:97] "Volume reconstruction finished" May 20 13:50:15.731317 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.731277 2575 reconciler.go:26] "Reconciler: start to sync state" May 20 13:50:15.731914 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.731862 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:15.731914 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.730793 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18b14a4b7439e40d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-05-20 13:50:15.721010189 +0000 UTC m=+0.471822576,LastTimestamp:2026-05-20 13:50:15.721010189 +0000 UTC m=+0.471822576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" May 20 13:50:15.732342 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732211 2575 factory.go:55] Registering systemd factory May 20 13:50:15.732342 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732223 2575 factory.go:223] Registration of the systemd container factory successfully May 20 13:50:15.732570 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732482 2575 factory.go:153] Registering CRI-O factory May 20 13:50:15.732570 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732498 2575 factory.go:223] Registration of the crio container factory successfully May 20 13:50:15.732657 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732570 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 20 13:50:15.732657 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732595 2575 factory.go:103] Registering Raw factory May 20 13:50:15.732733 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.732669 2575 manager.go:1196] Started watching for new ooms in manager May 20 13:50:15.734144 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.734127 2575 manager.go:319] Starting recovery of all containers May 20 13:50:15.741842 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.741803 2575 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 20 13:50:15.741966 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.741836 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 20 13:50:15.746181 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.746037 2575 manager.go:324] Recovery completed May 20 13:50:15.750835 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.750819 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.754994 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.754974 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.755067 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.755008 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.755067 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.755020 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.755651 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.755632 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" May 20 13:50:15.755651 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.755651 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 20 13:50:15.755786 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.755673 2575 state_mem.go:36] "Initialized new in-memory state store" May 20 13:50:15.758153 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.758079 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18b14a4b76407561 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-05-20 13:50:15.754995041 +0000 UTC m=+0.505807429,LastTimestamp:2026-05-20 13:50:15.754995041 +0000 UTC m=+0.505807429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" May 20 13:50:15.759280 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.759264 2575 policy_none.go:49] "None policy: Start" May 20 13:50:15.759352 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.759287 2575 memory_manager.go:186] "Starting memorymanager" policy="None" May 20 13:50:15.759352 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.759303 2575 state_mem.go:35] "Initializing new in-memory state store" May 20 13:50:15.765558 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.765538 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xgp7f" May 20 13:50:15.775678 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.775596 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18b14a4b7640bd64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-148.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-05-20 13:50:15.755013476 +0000 UTC m=+0.505825864,LastTimestamp:2026-05-20 13:50:15.755013476 +0000 UTC m=+0.505825864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" May 20 13:50:15.778947 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.778928 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xgp7f" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800340 2575 manager.go:341] "Starting Device Plugin manager" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.800381 2575 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800391 2575 server.go:85] "Starting device plugin registration server" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800639 2575 eviction_manager.go:189] "Eviction manager: starting control loop" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800651 2575 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800784 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800886 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.800897 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.801465 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 20 13:50:15.814347 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.801499 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:15.836627 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.836593 2575 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 20 13:50:15.837721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.837700 2575 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 20 13:50:15.837721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.837719 2575 status_manager.go:230] "Starting to sync pod status with apiserver" May 20 13:50:15.837856 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.837764 2575 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 20 13:50:15.837856 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.837775 2575 kubelet.go:2451] "Starting kubelet main sync loop" May 20 13:50:15.837856 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.837821 2575 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 20 13:50:15.840729 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.840709 2575 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:15.902264 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.902232 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.903269 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.903253 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.903362 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.903287 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.903362 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.903301 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.903362 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.903325 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-148.ec2.internal" May 20 13:50:15.911716 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.911702 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-148.ec2.internal" May 20 13:50:15.911784 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.911725 2575 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-148.ec2.internal\": node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:15.934665 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.934633 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:15.938696 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.938677 2575 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal"] May 20 13:50:15.938776 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.938765 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.941510 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.941488 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.941607 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.941517 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.941607 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.941532 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.943887 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.943873 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.944020 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.944006 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:15.944058 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.944037 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.947612 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947597 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.947703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947610 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.947703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947624 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.947703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947636 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.947703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947636 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.947893 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.947707 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.949967 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.949951 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:15.950032 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.949977 2575 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:15.950712 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.950696 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:15.950828 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.950720 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:15.950828 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:15.950733 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:15.979566 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.979543 2575 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.ec2.internal\" not found" node="ip-10-0-137-148.ec2.internal" May 20 13:50:15.984153 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:15.984135 2575 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.ec2.internal\" not found" node="ip-10-0-137-148.ec2.internal" May 20 13:50:16.033484 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.033451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.033484 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.033485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.033658 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.033527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18cf8ee325eb0038c1d0d88dc7ce7795-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"18cf8ee325eb0038c1d0d88dc7ce7795\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:16.035552 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.035535 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.134632 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134596 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.134781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134650 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.134781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134669 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18cf8ee325eb0038c1d0d88dc7ce7795-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"18cf8ee325eb0038c1d0d88dc7ce7795\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:16.134781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.134781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134727 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18cf8ee325eb0038c1d0d88dc7ce7795-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"18cf8ee325eb0038c1d0d88dc7ce7795\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:16.134781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.134737 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ff499fad8e2c7ab5638b55bcb9861e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"e0ff499fad8e2c7ab5638b55bcb9861e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.135613 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.135596 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.236087 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.236006 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.282193 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.282157 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:16.286900 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.286875 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:16.336841 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.336805 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.437302 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.437274 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.537832 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.537718 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.624965 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.624934 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 20 13:50:16.625455 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.625103 2575 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 20 13:50:16.638326 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.638300 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.730585 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.730557 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 20 13:50:16.739224 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.739196 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.756958 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.756929 2575 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 20 13:50:16.778600 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.778566 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwxgk" May 20 13:50:16.780626 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.780590 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:15 +0000 UTC" deadline="2027-10-20 07:59:56.196154925 +0000 UTC" May 20 13:50:16.780768 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.780627 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12426h9m39.41553125s" May 20 13:50:16.785729 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.785708 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwxgk" May 20 13:50:16.839358 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.839276 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.865107 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.865078 2575 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:16.917723 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:16.917673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ff499fad8e2c7ab5638b55bcb9861e.slice/crio-159af087fe5595081e81dd91b933c33cc335807216a8831920def924d3ffd614 WatchSource:0}: Error finding container 159af087fe5595081e81dd91b933c33cc335807216a8831920def924d3ffd614: Status 404 returned error can't find the container with id 159af087fe5595081e81dd91b933c33cc335807216a8831920def924d3ffd614 May 20 13:50:16.917983 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:16.917962 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cf8ee325eb0038c1d0d88dc7ce7795.slice/crio-af80fdac5a0783cd713583d1567287a62b33a53c7f0d7c295e8571db693333f9 WatchSource:0}: Error finding container af80fdac5a0783cd713583d1567287a62b33a53c7f0d7c295e8571db693333f9: Status 404 returned error can't find the container with id af80fdac5a0783cd713583d1567287a62b33a53c7f0d7c295e8571db693333f9 May 20 13:50:16.922525 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.922508 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 13:50:16.940144 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:16.940116 2575 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" May 20 13:50:16.969650 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:16.969621 2575 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:17.032091 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.032063 2575 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" May 20 13:50:17.065949 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.065919 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 20 13:50:17.067697 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.067682 2575 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" May 20 13:50:17.081478 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.081454 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 20 13:50:17.200695 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.200653 2575 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:17.706103 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.706068 2575 apiserver.go:52] "Watching apiserver" May 20 13:50:17.716538 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.716501 2575 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 20 13:50:17.718422 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.718387 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lfcgh","kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9mtzp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal","openshift-multus/multus-additional-cni-plugins-z6nqd","openshift-multus/multus-xlk7j","openshift-multus/network-metrics-daemon-5pjpk","openshift-ovn-kubernetes/ovnkube-node-lhfql","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw","openshift-image-registry/node-ca-nknqs","openshift-network-diagnostics/network-check-target-dlhw8","openshift-network-operator/iptables-alerter-75246"] May 20 13:50:17.721159 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.721128 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.723452 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.723429 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.725826 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.725650 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.728201 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.727881 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:17.728201 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.728031 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:17.729140 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.728535 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 20 13:50:17.729140 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.728873 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 20 13:50:17.729140 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.729074 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qspn5\"" May 20 13:50:17.729831 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.729807 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 20 13:50:17.729831 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.729825 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 20 13:50:17.730022 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.729979 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dlgjv\"" May 20 13:50:17.730117 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.730042 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.730117 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.730063 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 20 13:50:17.730221 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.730202 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 20 13:50:17.730616 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.730595 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 20 13:50:17.731118 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.731096 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnfhd\"" May 20 13:50:17.731205 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.731181 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 20 13:50:17.734597 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.734575 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 20 13:50:17.734820 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.734798 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.734914 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.734819 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 20 13:50:17.734914 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.734902 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlk7j" May 20 13:50:17.737417 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.737396 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.737516 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.737493 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:17.739950 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.739921 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.741572 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:17.741675 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.741675 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-var-lib-kubelet\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.741675 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-host\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.741852 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-tuned\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.741852 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-kubernetes\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.741852 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cnibin\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.741852 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-os-release\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.741852 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-sys\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-binary-copy\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-run\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-registration-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.741970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-modprobe-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-socket-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742084 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/311a8278-e305-4346-8c23-268e7cbd047f-kube-api-access-cx9gs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-system-cni-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-systemd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-lib-modules\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742189 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysconfig\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-device-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-sys-fs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-konnectivity-ca\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.742449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgfm\" (UniqueName: \"kubernetes.io/projected/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-kube-api-access-msgfm\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-conf\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25c12a44-9f17-44ff-b915-c31404a6871e-serviceca\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrs6\" (UniqueName: \"kubernetes.io/projected/25c12a44-9f17-44ff-b915-c31404a6871e-kube-api-access-dwrs6\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-agent-certs\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-tmp\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25c12a44-9f17-44ff-b915-c31404a6871e-host\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rrd\" (UniqueName: \"kubernetes.io/projected/e5fad16b-1776-49b5-b184-73ca45bb3339-kube-api-access-g6rrd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.743018 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.742988 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7jcz2\"" May 20 13:50:17.743383 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.743181 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 20 13:50:17.743383 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.743342 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 20 13:50:17.743927 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.743887 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pdwk9\"" May 20 13:50:17.744103 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.743989 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-96zns\"" May 20 13:50:17.744103 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.744066 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.745439 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.745628 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746019 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746050 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746117 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746356 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746399 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746553 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746607 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 20 13:50:17.746857 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.746722 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 20 13:50:17.747334 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.747065 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 20 13:50:17.747449 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.747430 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 20 13:50:17.747881 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.747859 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gvkl\"" May 20 13:50:17.751294 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.751271 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mkxvw\"" May 20 13:50:17.786773 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.786726 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:16 +0000 UTC" deadline="2027-12-16 08:52:48.35006911 +0000 UTC" May 20 13:50:17.786773 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.786765 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13795h2m30.563306994s" May 20 13:50:17.831973 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.831946 2575 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 20 13:50:17.842193 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.842142 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"e0ff499fad8e2c7ab5638b55bcb9861e","Type":"ContainerStarted","Data":"159af087fe5595081e81dd91b933c33cc335807216a8831920def924d3ffd614"} May 20 13:50:17.843114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843087 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-device-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843122 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-sys-fs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843147 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-msgfm\" (UniqueName: \"kubernetes.io/projected/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-kube-api-access-msgfm\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktsb\" (UniqueName: \"kubernetes.io/projected/2a5587d1-e164-4009-995a-e55a34ad59ab-kube-api-access-kktsb\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843183 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" event={"ID":"18cf8ee325eb0038c1d0d88dc7ce7795","Type":"ContainerStarted","Data":"af80fdac5a0783cd713583d1567287a62b33a53c7f0d7c295e8571db693333f9"} May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-system-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843236 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrs6\" (UniqueName: \"kubernetes.io/projected/25c12a44-9f17-44ff-b915-c31404a6871e-kube-api-access-dwrs6\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.843245 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843240 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-device-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-sys-fs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843262 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-agent-certs\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-var-lib-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-bin\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-socket-dir-parent\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-kubelet\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843397 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25c12a44-9f17-44ff-b915-c31404a6871e-host\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843431 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rrd\" (UniqueName: \"kubernetes.io/projected/e5fad16b-1776-49b5-b184-73ca45bb3339-kube-api-access-g6rrd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-script-lib\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-k8s-cni-cncf-io\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.843548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-bin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-etc-kubernetes\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843560 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25c12a44-9f17-44ff-b915-c31404a6871e-host\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843604 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843636 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843638 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843693 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-tuned\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-systemd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-env-overrides\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843811 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843862 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-kubernetes\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843893 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cnibin\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843918 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-os-release\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843943 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-sys\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-kubernetes\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-slash\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.843952 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cnibin\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-hostroot\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844022 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-sys\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844037 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-os-release\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844043 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-binary-copy\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844072 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-run\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29twt\" (UniqueName: \"kubernetes.io/projected/b8a9e50a-d38e-4123-9a51-fde818feb241-kube-api-access-29twt\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844164 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-run\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-netd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844199 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-registration-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844216 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844299 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-registration-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844300 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844340 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-etc-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5587d1-e164-4009-995a-e55a34ad59ab-host-slash\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.844924 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844439 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-socket-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844465 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-system-cni-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844482 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-systemd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-socket-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-systemd-units\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-cnibin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-system-cni-dir\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-multus-certs\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9qx\" (UniqueName: \"kubernetes.io/projected/821307da-511e-49b1-be2f-7c58b70de56d-kube-api-access-wb9qx\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844690 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysconfig\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-systemd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844707 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844727 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-konnectivity-ca\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844734 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-binary-copy\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysconfig\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844841 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.845659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844848 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844879 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-conf\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-netns\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844933 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25c12a44-9f17-44ff-b915-c31404a6871e-serviceca\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-sysctl-conf\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.844998 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-tmp\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-node-log\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5587d1-e164-4009-995a-e55a34ad59ab-iptables-alerter-script\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-multus\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-os-release\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-multus-daemon-config\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845237 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-var-lib-kubelet\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845277 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-host\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-conf-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845312 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-var-lib-kubelet\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-config\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845346 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-konnectivity-ca\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-host\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7fc\" (UniqueName: \"kubernetes.io/projected/6e9f8fc8-87cf-49f2-82f7-4232a3698889-kube-api-access-vq7fc\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845426 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-ovn\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-log-socket\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25c12a44-9f17-44ff-b915-c31404a6871e-serviceca\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845511 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-modprobe-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-netns\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845558 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845581 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/311a8278-e305-4346-8c23-268e7cbd047f-kube-api-access-cx9gs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845683 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-lib-modules\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845685 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-modprobe-d\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311a8278-e305-4346-8c23-268e7cbd047f-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-kubelet\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovn-node-metrics-cert\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.846895 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-cni-binary-copy\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.847592 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.845795 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fad16b-1776-49b5-b184-73ca45bb3339-lib-modules\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.847592 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.847452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-tmp\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.847592 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.847476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5fad16b-1776-49b5-b184-73ca45bb3339-etc-tuned\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.847761 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.847725 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd716f72-80d8-4c8e-bfed-6f3053ad0a3b-agent-certs\") pod \"konnectivity-agent-lfcgh\" (UID: \"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b\") " pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:17.866379 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.866345 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:17.866379 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.866379 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:17.866601 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.866393 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:17.866601 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.866480 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:18.366442618 +0000 UTC m=+3.117255007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:17.868138 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.868104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrs6\" (UniqueName: \"kubernetes.io/projected/25c12a44-9f17-44ff-b915-c31404a6871e-kube-api-access-dwrs6\") pod \"node-ca-nknqs\" (UID: \"25c12a44-9f17-44ff-b915-c31404a6871e\") " pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:17.868240 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.868146 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/311a8278-e305-4346-8c23-268e7cbd047f-kube-api-access-cx9gs\") pod \"aws-ebs-csi-driver-node-kpvpw\" (UID: \"311a8278-e305-4346-8c23-268e7cbd047f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:17.868555 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.868530 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgfm\" (UniqueName: \"kubernetes.io/projected/d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa-kube-api-access-msgfm\") pod \"multus-additional-cni-plugins-z6nqd\" (UID: \"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa\") " pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:17.869003 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.868983 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rrd\" (UniqueName: \"kubernetes.io/projected/e5fad16b-1776-49b5-b184-73ca45bb3339-kube-api-access-g6rrd\") pod \"tuned-9mtzp\" (UID: \"e5fad16b-1776-49b5-b184-73ca45bb3339\") " pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:17.946154 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946116 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-slash\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946154 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946157 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-hostroot\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946176 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29twt\" (UniqueName: \"kubernetes.io/projected/b8a9e50a-d38e-4123-9a51-fde818feb241-kube-api-access-29twt\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946204 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946226 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-netd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946244 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946248 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-slash\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-hostroot\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946312 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-netd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946261 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-etc-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946386 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5587d1-e164-4009-995a-e55a34ad59ab-host-slash\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.946402 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.946404 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946416 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-systemd-units\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946445 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-cnibin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946482 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-multus-certs\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946512 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9qx\" (UniqueName: \"kubernetes.io/projected/821307da-511e-49b1-be2f-7c58b70de56d-kube-api-access-wb9qx\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:17.946530 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:18.446514336 +0000 UTC m=+3.197326712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946551 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946571 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-netns\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946599 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-node-log\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-systemd-units\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-etc-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946627 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946662 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-cnibin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946675 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5587d1-e164-4009-995a-e55a34ad59ab-iptables-alerter-script\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946701 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-multus-certs\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946706 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-multus\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946730 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-os-release\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.946984 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946768 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-multus-daemon-config\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946792 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-conf-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946816 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-config\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946841 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7fc\" (UniqueName: \"kubernetes.io/projected/6e9f8fc8-87cf-49f2-82f7-4232a3698889-kube-api-access-vq7fc\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946876 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-ovn\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946901 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-log-socket\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946939 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-netns\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946976 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-kubelet\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947003 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovn-node-metrics-cert\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947028 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-cni-binary-copy\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947027 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947053 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kktsb\" (UniqueName: \"kubernetes.io/projected/2a5587d1-e164-4009-995a-e55a34ad59ab-kube-api-access-kktsb\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947075 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-system-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947090 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-netns\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.946569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5587d1-e164-4009-995a-e55a34ad59ab-host-slash\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947107 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-var-lib-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947127 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-bin\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.947783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947143 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-socket-dir-parent\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947167 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-kubelet\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-node-log\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947201 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-script-lib\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-multus\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947221 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-k8s-cni-cncf-io\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947237 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-bin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5587d1-e164-4009-995a-e55a34ad59ab-iptables-alerter-script\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947274 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-os-release\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947285 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-etc-kubernetes\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947256 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-etc-kubernetes\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-kubelet\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947337 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-systemd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947360 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-env-overrides\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947392 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947474 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-kubelet\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.948904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-multus-daemon-config\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-ovn\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.947973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-log-socket\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-run-netns\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948058 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-var-lib-openvswitch\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-host-cni-bin\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-system-cni-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-socket-dir-parent\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-var-lib-cni-bin\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e9f8fc8-87cf-49f2-82f7-4232a3698889-run-systemd\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-script-lib\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948287 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-host-run-k8s-cni-cncf-io\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821307da-511e-49b1-be2f-7c58b70de56d-multus-conf-dir\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948474 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovnkube-config\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821307da-511e-49b1-be2f-7c58b70de56d-cni-binary-copy\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.949617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.948609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e9f8fc8-87cf-49f2-82f7-4232a3698889-env-overrides\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.950242 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.949807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e9f8fc8-87cf-49f2-82f7-4232a3698889-ovn-node-metrics-cert\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:17.959475 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.959410 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29twt\" (UniqueName: \"kubernetes.io/projected/b8a9e50a-d38e-4123-9a51-fde818feb241-kube-api-access-29twt\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:17.961937 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.961914 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktsb\" (UniqueName: \"kubernetes.io/projected/2a5587d1-e164-4009-995a-e55a34ad59ab-kube-api-access-kktsb\") pod \"iptables-alerter-75246\" (UID: \"2a5587d1-e164-4009-995a-e55a34ad59ab\") " pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:17.962593 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.962569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9qx\" (UniqueName: \"kubernetes.io/projected/821307da-511e-49b1-be2f-7c58b70de56d-kube-api-access-wb9qx\") pod \"multus-xlk7j\" (UID: \"821307da-511e-49b1-be2f-7c58b70de56d\") " pod="openshift-multus/multus-xlk7j" May 20 13:50:17.963861 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:17.963845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7fc\" (UniqueName: \"kubernetes.io/projected/6e9f8fc8-87cf-49f2-82f7-4232a3698889-kube-api-access-vq7fc\") pod \"ovnkube-node-lhfql\" (UID: \"6e9f8fc8-87cf-49f2-82f7-4232a3698889\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:18.005432 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.005400 2575 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:18.033812 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.033777 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" May 20 13:50:18.041633 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.041606 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" May 20 13:50:18.052454 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.052421 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nknqs" May 20 13:50:18.058849 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.058813 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:18.067539 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.067504 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" May 20 13:50:18.075309 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.075283 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlk7j" May 20 13:50:18.080078 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.080051 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:18.086850 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.086821 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75246" May 20 13:50:18.450607 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.450566 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:18.450823 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.450662 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:18.450823 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450769 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:18.450823 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450818 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:18.450990 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450837 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:18.450990 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450846 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:19.450825754 +0000 UTC m=+4.201638128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:18.450990 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450853 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:18.450990 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:18.450907 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:19.450891495 +0000 UTC m=+4.201703888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:18.508122 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.508084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c12a44_9f17_44ff_b915_c31404a6871e.slice/crio-5b171a8f606c09f5835bc87bb06dce61753872a59f20fa756c851ed40d4c020f WatchSource:0}: Error finding container 5b171a8f606c09f5835bc87bb06dce61753872a59f20fa756c851ed40d4c020f: Status 404 returned error can't find the container with id 5b171a8f606c09f5835bc87bb06dce61753872a59f20fa756c851ed40d4c020f May 20 13:50:18.512088 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.512064 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd716f72_80d8_4c8e_bfed_6f3053ad0a3b.slice/crio-11f68fa464cc090b2166c630a39d34b560dc79bc4ecb3b0a5690057110892689 WatchSource:0}: Error finding container 11f68fa464cc090b2166c630a39d34b560dc79bc4ecb3b0a5690057110892689: Status 404 returned error can't find the container with id 11f68fa464cc090b2166c630a39d34b560dc79bc4ecb3b0a5690057110892689 May 20 13:50:18.513020 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.512972 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fad16b_1776_49b5_b184_73ca45bb3339.slice/crio-3fc1b40323bc1a690bd3f634838dcbaa580c7044f59a83bc333e0161151a81ae WatchSource:0}: Error finding container 3fc1b40323bc1a690bd3f634838dcbaa580c7044f59a83bc333e0161151a81ae: Status 404 returned error can't find the container with id 3fc1b40323bc1a690bd3f634838dcbaa580c7044f59a83bc333e0161151a81ae May 20 13:50:18.514243 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.514220 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311a8278_e305_4346_8c23_268e7cbd047f.slice/crio-e56c7943e67a95ebcb5cbfe25fda6c590bcb70733aa035b7c8add906acf2bbf3 WatchSource:0}: Error finding container e56c7943e67a95ebcb5cbfe25fda6c590bcb70733aa035b7c8add906acf2bbf3: Status 404 returned error can't find the container with id e56c7943e67a95ebcb5cbfe25fda6c590bcb70733aa035b7c8add906acf2bbf3 May 20 13:50:18.515231 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.515204 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821307da_511e_49b1_be2f_7c58b70de56d.slice/crio-aaa110e33813987131a46985567b2faef4bf12e0a41136564794a2736c8448c1 WatchSource:0}: Error finding container aaa110e33813987131a46985567b2faef4bf12e0a41136564794a2736c8448c1: Status 404 returned error can't find the container with id aaa110e33813987131a46985567b2faef4bf12e0a41136564794a2736c8448c1 May 20 13:50:18.516809 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.516336 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ca5fdc_47a4_4a7c_b0df_16857b6f9dfa.slice/crio-85dd15d09955889a87a5fc0d2aa979203b4411bc66f3c0e36a51f2b23487f492 WatchSource:0}: Error finding container 85dd15d09955889a87a5fc0d2aa979203b4411bc66f3c0e36a51f2b23487f492: Status 404 returned error can't find the container with id 85dd15d09955889a87a5fc0d2aa979203b4411bc66f3c0e36a51f2b23487f492 May 20 13:50:18.517487 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.517054 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9f8fc8_87cf_49f2_82f7_4232a3698889.slice/crio-d9e960ab778fb47e3f6bb30dc2b3aadbd6e30cc9d1ac88e9f8768429ff16738e WatchSource:0}: Error finding container d9e960ab778fb47e3f6bb30dc2b3aadbd6e30cc9d1ac88e9f8768429ff16738e: Status 404 returned error can't find the container with id d9e960ab778fb47e3f6bb30dc2b3aadbd6e30cc9d1ac88e9f8768429ff16738e May 20 13:50:18.518559 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:18.518455 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5587d1_e164_4009_995a_e55a34ad59ab.slice/crio-58eb810290c11328ddedd572187aadd0ff2dcf078bcf8ef0475e4881b978f646 WatchSource:0}: Error finding container 58eb810290c11328ddedd572187aadd0ff2dcf078bcf8ef0475e4881b978f646: Status 404 returned error can't find the container with id 58eb810290c11328ddedd572187aadd0ff2dcf078bcf8ef0475e4881b978f646 May 20 13:50:18.787891 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.787700 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:16 +0000 UTC" deadline="2027-12-17 12:07:30.513242765 +0000 UTC" May 20 13:50:18.787891 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.787891 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13822h17m11.725356984s" May 20 13:50:18.846907 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.846866 2575 generic.go:358] "Generic (PLEG): container finished" podID="e0ff499fad8e2c7ab5638b55bcb9861e" containerID="77e22cffc8f07f9dd3bb8745ed7a396cc909701f55687872edec9b025eb7ede1" exitCode=0 May 20 13:50:18.847076 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.846952 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"e0ff499fad8e2c7ab5638b55bcb9861e","Type":"ContainerDied","Data":"77e22cffc8f07f9dd3bb8745ed7a396cc909701f55687872edec9b025eb7ede1"} May 20 13:50:18.848078 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.848041 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"d9e960ab778fb47e3f6bb30dc2b3aadbd6e30cc9d1ac88e9f8768429ff16738e"} May 20 13:50:18.849445 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.849408 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" event={"ID":"e5fad16b-1776-49b5-b184-73ca45bb3339","Type":"ContainerStarted","Data":"3fc1b40323bc1a690bd3f634838dcbaa580c7044f59a83bc333e0161151a81ae"} May 20 13:50:18.850439 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.850404 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerStarted","Data":"85dd15d09955889a87a5fc0d2aa979203b4411bc66f3c0e36a51f2b23487f492"} May 20 13:50:18.851442 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.851415 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlk7j" event={"ID":"821307da-511e-49b1-be2f-7c58b70de56d","Type":"ContainerStarted","Data":"aaa110e33813987131a46985567b2faef4bf12e0a41136564794a2736c8448c1"} May 20 13:50:18.852579 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.852514 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" event={"ID":"311a8278-e305-4346-8c23-268e7cbd047f","Type":"ContainerStarted","Data":"e56c7943e67a95ebcb5cbfe25fda6c590bcb70733aa035b7c8add906acf2bbf3"} May 20 13:50:18.854054 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.854021 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" event={"ID":"18cf8ee325eb0038c1d0d88dc7ce7795","Type":"ContainerStarted","Data":"9d80eb3b6e7f4e0c618a79b4b616e3d0ad6a6c00061658cf879093d6b8f7d65b"} May 20 13:50:18.855312 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.855290 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75246" event={"ID":"2a5587d1-e164-4009-995a-e55a34ad59ab","Type":"ContainerStarted","Data":"58eb810290c11328ddedd572187aadd0ff2dcf078bcf8ef0475e4881b978f646"} May 20 13:50:18.856369 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.856338 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lfcgh" event={"ID":"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b","Type":"ContainerStarted","Data":"11f68fa464cc090b2166c630a39d34b560dc79bc4ecb3b0a5690057110892689"} May 20 13:50:18.857474 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:18.857452 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nknqs" event={"ID":"25c12a44-9f17-44ff-b915-c31404a6871e","Type":"ContainerStarted","Data":"5b171a8f606c09f5835bc87bb06dce61753872a59f20fa756c851ed40d4c020f"} May 20 13:50:19.459953 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.459910 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:19.460152 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.459974 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:19.460152 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460090 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:19.460152 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460096 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:19.460152 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460126 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:19.460152 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460139 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:19.460461 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460162 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:21.460141653 +0000 UTC m=+6.210954041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:19.460461 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.460193 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:21.460173414 +0000 UTC m=+6.210985793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:19.839052 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.838971 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:19.839519 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.839124 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:19.839629 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.839610 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:19.839778 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:19.839719 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:19.871664 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.871625 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"e0ff499fad8e2c7ab5638b55bcb9861e","Type":"ContainerStarted","Data":"c416741e1e55a3981f8b3bf45fddad92f5d3bd45bc73baea911e2a25889e514c"} May 20 13:50:19.893798 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.893710 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" podStartSLOduration=2.893688902 podStartE2EDuration="2.893688902s" podCreationTimestamp="2026-05-20 13:50:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:18.912941906 +0000 UTC m=+3.663754304" watchObservedRunningTime="2026-05-20 13:50:19.893688902 +0000 UTC m=+4.644501300" May 20 13:50:19.893995 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:19.893940 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" podStartSLOduration=2.893928169 podStartE2EDuration="2.893928169s" podCreationTimestamp="2026-05-20 13:50:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:19.893434008 +0000 UTC m=+4.644246408" watchObservedRunningTime="2026-05-20 13:50:19.893928169 +0000 UTC m=+4.644740568" May 20 13:50:21.478951 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:21.478895 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:21.478962 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479100 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479170 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.479149732 +0000 UTC m=+10.229962124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479311 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479338 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479351 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:21.479429 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.479411 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.479395136 +0000 UTC m=+10.230207514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:21.840668 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:21.840492 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:21.840668 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.840640 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:21.841100 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:21.841078 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:21.841202 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:21.841180 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:23.839135 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:23.838379 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:23.839135 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:23.838481 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:23.839135 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:23.838925 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:23.839135 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:23.839091 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:25.512175 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:25.512130 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:25.512201 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512339 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512410 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:33.512387745 +0000 UTC m=+18.263200135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512506 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512533 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512549 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:25.512721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.512605 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:33.512587694 +0000 UTC m=+18.263400085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:25.839968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:25.839875 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:25.839968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:25.839936 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:25.840165 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.840062 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:25.840217 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:25.840174 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:27.838266 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:27.838229 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:27.838707 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:27.838242 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:27.838707 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:27.838362 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:27.838707 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:27.838394 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:28.362736 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.362696 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tzl92"] May 20 13:50:28.377639 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.377604 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.380662 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.380630 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 20 13:50:28.380662 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.380655 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntfbn\"" May 20 13:50:28.381496 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.381476 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 20 13:50:28.436848 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.436811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c70d7c-144e-473d-9344-f2947744cada-tmp-dir\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.437041 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.436878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2d45\" (UniqueName: \"kubernetes.io/projected/e3c70d7c-144e-473d-9344-f2947744cada-kube-api-access-s2d45\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.437041 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.437002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c70d7c-144e-473d-9344-f2947744cada-hosts-file\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.537825 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.537722 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c70d7c-144e-473d-9344-f2947744cada-tmp-dir\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.537825 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.537808 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s2d45\" (UniqueName: \"kubernetes.io/projected/e3c70d7c-144e-473d-9344-f2947744cada-kube-api-access-s2d45\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.538011 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.537873 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c70d7c-144e-473d-9344-f2947744cada-hosts-file\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.538011 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.537952 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c70d7c-144e-473d-9344-f2947744cada-hosts-file\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.538114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.538093 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c70d7c-144e-473d-9344-f2947744cada-tmp-dir\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.548845 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.548806 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2d45\" (UniqueName: \"kubernetes.io/projected/e3c70d7c-144e-473d-9344-f2947744cada-kube-api-access-s2d45\") pod \"node-resolver-tzl92\" (UID: \"e3c70d7c-144e-473d-9344-f2947744cada\") " pod="openshift-dns/node-resolver-tzl92" May 20 13:50:28.688095 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:28.688055 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tzl92" May 20 13:50:29.838780 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:29.838726 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:29.839320 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:29.838893 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:29.839383 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:29.839327 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:29.839458 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:29.839430 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:31.838280 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:31.838246 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:31.838619 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:31.838257 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:31.838619 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:31.838373 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:31.838619 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:31.838422 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:33.574808 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.574763 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.574832 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.574915 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.574943 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.574945 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.574957 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.575020 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.574998814 +0000 UTC m=+34.325811207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:33.575324 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.575068 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.575057252 +0000 UTC m=+34.325869627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:33.803140 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.803103 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kdngv"] May 20 13:50:33.817371 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.817336 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.817531 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.817443 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:33.838072 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.837981 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:33.838222 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.838095 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:33.838222 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.838156 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:33.838340 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.838237 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:33.877071 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.877034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-kubelet-config\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.877258 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.877079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-dbus\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.877258 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.877105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.977371 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-dbus\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977599 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.977424 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977599 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.977595 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-kubelet-config\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977712 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.977604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-dbus\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977712 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.977647 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:33.977712 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:33.977680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/376a9486-73e2-4fe4-b9a1-57419dc0c808-kubelet-config\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:33.977873 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:33.977726 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:34.477705653 +0000 UTC m=+19.228518053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:34.481131 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:34.481088 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:34.481340 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:34.481264 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:34.481340 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:34.481338 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:35.481322023 +0000 UTC m=+20.232134403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:35.488639 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:35.488596 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:35.489079 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:35.488717 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:35.489079 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:35.488795 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:37.4887797 +0000 UTC m=+22.239592075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:35.838808 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:35.838722 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:35.838966 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:35.838844 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:35.838966 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:35.838859 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:35.838966 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:35.838840 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:35.838966 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:35.838944 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:35.839148 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:35.839031 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:36.366877 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:50:36.366848 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c70d7c_144e_473d_9344_f2947744cada.slice/crio-8aacc3a122d0b77e4b259d34cca662930aa5cbc24509b188020f668c8469c0fd WatchSource:0}: Error finding container 8aacc3a122d0b77e4b259d34cca662930aa5cbc24509b188020f668c8469c0fd: Status 404 returned error can't find the container with id 8aacc3a122d0b77e4b259d34cca662930aa5cbc24509b188020f668c8469c0fd May 20 13:50:36.904190 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.903827 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" event={"ID":"311a8278-e305-4346-8c23-268e7cbd047f","Type":"ContainerStarted","Data":"4ebb7f2c324c99aa709616c1aa9edc6f3e180fe828adc5c31e2e9d43e173bf03"} May 20 13:50:36.905281 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.905249 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tzl92" event={"ID":"e3c70d7c-144e-473d-9344-f2947744cada","Type":"ContainerStarted","Data":"a1e233a18b559c2b36b6b27bf1d811e46a90992c1c9ee41c9a6451922f5ddd59"} May 20 13:50:36.905381 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.905297 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tzl92" event={"ID":"e3c70d7c-144e-473d-9344-f2947744cada","Type":"ContainerStarted","Data":"8aacc3a122d0b77e4b259d34cca662930aa5cbc24509b188020f668c8469c0fd"} May 20 13:50:36.906547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.906524 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lfcgh" event={"ID":"fd716f72-80d8-4c8e-bfed-6f3053ad0a3b","Type":"ContainerStarted","Data":"33de4bb992d58a680b6318b3f38b6068a77657fa5ebe8a4eaa45e0fa6e72dadc"} May 20 13:50:36.907804 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.907780 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nknqs" event={"ID":"25c12a44-9f17-44ff-b915-c31404a6871e","Type":"ContainerStarted","Data":"09daf1f4884d0f43f3baf83240416e914cbc28081712ad7320df340f2957c8ce"} May 20 13:50:36.909964 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.909942 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"5e6217149a547342cbb4859b61e5839e138f29ef865c5bca14211502924d9f92"} May 20 13:50:36.910042 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.909970 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"f825ef918c627301ac1f15d335839708ecdd14d6defba9e720523c477c8680f9"} May 20 13:50:36.910042 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.909981 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"f0aa167f5c1526e0bfa7a2bada523a330d7c4e75c9f135b6ca71c60204843f9e"} May 20 13:50:36.910042 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.909994 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"e42d14089b8c581a02f040a0c60ab040958e5c5e5fec77e1be3671b3571e6bb2"} May 20 13:50:36.910042 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.910002 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"5a7566a7b4a405872dccfc81c2d6db9842fe206ea75fa35c8ed1acb853d518fe"} May 20 13:50:36.911154 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.911133 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" event={"ID":"e5fad16b-1776-49b5-b184-73ca45bb3339","Type":"ContainerStarted","Data":"8e048311abd7660ecb7574e1382868704bc1cae64b3da38e18746fff1ea1ea3c"} May 20 13:50:36.912363 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.912342 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="2747522bdc1a1a757bbc9c121bcd2441edc2496c92f4d2da23e16a089f1ecd18" exitCode=0 May 20 13:50:36.912442 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.912394 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"2747522bdc1a1a757bbc9c121bcd2441edc2496c92f4d2da23e16a089f1ecd18"} May 20 13:50:36.913584 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.913562 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlk7j" event={"ID":"821307da-511e-49b1-be2f-7c58b70de56d","Type":"ContainerStarted","Data":"1c04f2f1bae9a9b970700847659b4e66c027ba6347e3a01f5209baa4d0d1566e"} May 20 13:50:36.924342 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.924299 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-tzl92" podStartSLOduration=8.924286007 podStartE2EDuration="8.924286007s" podCreationTimestamp="2026-05-20 13:50:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:36.924122529 +0000 UTC m=+21.674934926" watchObservedRunningTime="2026-05-20 13:50:36.924286007 +0000 UTC m=+21.675098403" May 20 13:50:36.946876 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.946826 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9mtzp" podStartSLOduration=4.099507753 podStartE2EDuration="21.946811521s" podCreationTimestamp="2026-05-20 13:50:15 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.515347881 +0000 UTC m=+3.266160271" lastFinishedPulling="2026-05-20 13:50:36.362651648 +0000 UTC m=+21.113464039" observedRunningTime="2026-05-20 13:50:36.946393182 +0000 UTC m=+21.697205580" watchObservedRunningTime="2026-05-20 13:50:36.946811521 +0000 UTC m=+21.697623961" May 20 13:50:36.963572 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:36.963488 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lfcgh" podStartSLOduration=3.115903747 podStartE2EDuration="20.963475253s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.514255033 +0000 UTC m=+3.265067411" lastFinishedPulling="2026-05-20 13:50:36.361826537 +0000 UTC m=+21.112638917" observedRunningTime="2026-05-20 13:50:36.96341563 +0000 UTC m=+21.714228029" watchObservedRunningTime="2026-05-20 13:50:36.963475253 +0000 UTC m=+21.714287651" May 20 13:50:37.054673 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.054622 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nknqs" podStartSLOduration=8.964412003 podStartE2EDuration="22.054600006s" podCreationTimestamp="2026-05-20 13:50:15 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.511487619 +0000 UTC m=+3.262300008" lastFinishedPulling="2026-05-20 13:50:31.601675625 +0000 UTC m=+16.352488011" observedRunningTime="2026-05-20 13:50:37.011508913 +0000 UTC m=+21.762321311" watchObservedRunningTime="2026-05-20 13:50:37.054600006 +0000 UTC m=+21.805412404" May 20 13:50:37.504515 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.504474 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:37.504678 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:37.504602 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:37.504767 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:37.504677 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:41.504657965 +0000 UTC m=+26.255470358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:37.838782 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.838664 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:37.838947 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.838778 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:37.838947 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.838837 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:37.839062 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:37.838938 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:37.839241 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:37.839212 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:37.839403 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:37.839375 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:37.889163 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.889127 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 20 13:50:37.917036 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.916990 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" event={"ID":"311a8278-e305-4346-8c23-268e7cbd047f","Type":"ContainerStarted","Data":"d7824f22f8121dc91a3dc5ce88e0848734912cba5eaf4fa5ebd533577be8f2e9"} May 20 13:50:37.918571 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.918519 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75246" event={"ID":"2a5587d1-e164-4009-995a-e55a34ad59ab","Type":"ContainerStarted","Data":"464ac17be8f8a0d901303e8600d98a9923460f4744fd23da9b53f87e8b705a78"} May 20 13:50:37.921670 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.921597 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"d54ba6bd87de67ac8a1897f235c5278e3bde33259e21c2c27db0a6c5acc98981"} May 20 13:50:37.927144 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.927120 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:37.927791 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.927770 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:37.938019 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.937969 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-xlk7j" podStartSLOduration=4.034982985 podStartE2EDuration="21.937955512s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.517677826 +0000 UTC m=+3.268490215" lastFinishedPulling="2026-05-20 13:50:36.420650362 +0000 UTC m=+21.171462742" observedRunningTime="2026-05-20 13:50:37.054763424 +0000 UTC m=+21.805575815" watchObservedRunningTime="2026-05-20 13:50:37.937955512 +0000 UTC m=+22.688767903" May 20 13:50:37.938447 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:37.938412 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-75246" podStartSLOduration=4.097060402 podStartE2EDuration="21.938406407s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.520077748 +0000 UTC m=+3.270890127" lastFinishedPulling="2026-05-20 13:50:36.361423742 +0000 UTC m=+21.112236132" observedRunningTime="2026-05-20 13:50:37.937773158 +0000 UTC m=+22.688585557" watchObservedRunningTime="2026-05-20 13:50:37.938406407 +0000 UTC m=+22.689218798" May 20 13:50:38.810497 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:38.810391 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:37.889151418Z","UUID":"2a3bc282-b112-4b72-a35a-94c12f5e379c","Handler":null,"Name":"","Endpoint":""} May 20 13:50:38.813079 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:38.813051 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 20 13:50:38.813237 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:38.813088 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 20 13:50:38.924006 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:38.923950 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:38.924485 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:38.924377 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lfcgh" May 20 13:50:39.839243 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.839048 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:39.839443 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.839048 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:39.839443 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:39.839372 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:39.839443 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.839078 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:39.839443 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:39.839404 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:39.839645 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:39.839492 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:39.928096 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.928059 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" event={"ID":"311a8278-e305-4346-8c23-268e7cbd047f","Type":"ContainerStarted","Data":"c12b30b94c6614e9f6a78b6edb3815cccd4c7268293d4fc9c59248ac0b7a0646"} May 20 13:50:39.931129 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.931104 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"4d52152d9af8f6111860d3093210e81a25463bc3a357a4616718c14cdf1124da"} May 20 13:50:39.953370 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:39.953304 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpvpw" podStartSLOduration=4.497979291 podStartE2EDuration="24.953286528s" podCreationTimestamp="2026-05-20 13:50:15 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.516381011 +0000 UTC m=+3.267193386" lastFinishedPulling="2026-05-20 13:50:38.971688248 +0000 UTC m=+23.722500623" observedRunningTime="2026-05-20 13:50:39.952740273 +0000 UTC m=+24.703552682" watchObservedRunningTime="2026-05-20 13:50:39.953286528 +0000 UTC m=+24.704098925" May 20 13:50:41.537771 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.537504 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:41.537771 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:41.537640 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:41.538472 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:41.537809 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.537793155 +0000 UTC m=+34.288605530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:41.838032 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.837888 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:41.838032 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.837927 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:41.838032 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:41.837998 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:41.838258 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:41.838061 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:41.838258 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.838079 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:41.838258 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:41.838152 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:41.937713 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.937672 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" event={"ID":"6e9f8fc8-87cf-49f2-82f7-4232a3698889","Type":"ContainerStarted","Data":"8fe2791201bf8b2572fbd219118ec8f9d411ad86a45d94615dc723116164502f"} May 20 13:50:41.938048 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.938013 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:41.939438 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.939413 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="05b5c9fa7976695c0c795f8a3951a532d6a6b9161600d286a6d834b240e7749b" exitCode=0 May 20 13:50:41.939528 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.939469 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"05b5c9fa7976695c0c795f8a3951a532d6a6b9161600d286a6d834b240e7749b"} May 20 13:50:41.954410 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.954374 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:41.975626 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:41.975578 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" podStartSLOduration=8.066678615 podStartE2EDuration="25.975563789s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.518944963 +0000 UTC m=+3.269757341" lastFinishedPulling="2026-05-20 13:50:36.427830139 +0000 UTC m=+21.178642515" observedRunningTime="2026-05-20 13:50:41.974217417 +0000 UTC m=+26.725029815" watchObservedRunningTime="2026-05-20 13:50:41.975563789 +0000 UTC m=+26.726376203" May 20 13:50:42.942674 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:42.942626 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:42.942674 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:42.942680 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:42.959202 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:42.959170 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:50:43.240520 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.240428 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlhw8"] May 20 13:50:43.240671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.240556 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:43.240671 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:43.240644 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:43.244137 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.244107 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pjpk"] May 20 13:50:43.244293 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.244253 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:43.244408 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:43.244386 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:43.244944 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.244923 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kdngv"] May 20 13:50:43.245044 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.245031 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:43.245143 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:43.245124 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:43.945816 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.945784 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="d8b666841e952fe90179b0fc9d2e1424d7ad48ba3aa558de0e06bd038e9fd3af" exitCode=0 May 20 13:50:43.946193 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:43.945875 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"d8b666841e952fe90179b0fc9d2e1424d7ad48ba3aa558de0e06bd038e9fd3af"} May 20 13:50:44.838399 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:44.838361 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:44.838645 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:44.838361 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:44.838645 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:44.838475 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:44.838645 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:44.838546 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:44.838645 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:44.838374 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:44.838645 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:44.838625 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:45.951527 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:45.951338 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="b35b64f80b217322107a16cab7085895c9ef092557d0e0e6beae4147572076a6" exitCode=0 May 20 13:50:45.952026 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:45.951418 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"b35b64f80b217322107a16cab7085895c9ef092557d0e0e6beae4147572076a6"} May 20 13:50:46.838997 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:46.838953 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:46.839190 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:46.838954 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:46.839190 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:46.838954 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:46.839190 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:46.839101 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:46.839417 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:46.839207 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:46.839417 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:46.839293 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:48.838789 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:48.838733 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:48.839382 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:48.838797 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:48.839382 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:48.838740 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:48.839382 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:48.838885 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:48.839382 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:48.838975 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:48.839382 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:48.839025 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:49.598524 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:49.598487 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:49.598721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:49.598556 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:49.598721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:49.598602 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:49.598721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598648 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:49.598721 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598708 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598725 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598757 2575 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598762 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs podName:b8a9e50a-d38e-4123-9a51-fde818feb241 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.598724851 +0000 UTC m=+66.349537230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs") pod "network-metrics-daemon-5pjpk" (UID: "b8a9e50a-d38e-4123-9a51-fde818feb241") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598770 2575 projected.go:194] Error preparing data for projected volume kube-api-access-xwqsf for pod openshift-network-diagnostics/network-check-target-dlhw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598781 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:05.598773489 +0000 UTC m=+50.349585864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:50:49.598964 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:49.598820 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf podName:c4fb9a1a-e7e7-47f2-af50-76404e0aabfb nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.598805289 +0000 UTC m=+66.349617689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xwqsf" (UniqueName: "kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf") pod "network-check-target-dlhw8" (UID: "c4fb9a1a-e7e7-47f2-af50-76404e0aabfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:50.838159 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:50.838115 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:50.838665 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:50.838110 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:50.838665 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:50.838274 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:50.838665 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:50.838324 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:50.838665 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:50.838348 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:50.838665 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:50.838434 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:51.965705 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:51.965671 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerStarted","Data":"2c398278800bc8e9ff53d51ea6235a0888173e0ebae6c5fa4d4a20a0a43f444e"} May 20 13:50:52.838572 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:52.838530 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:52.838733 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:52.838636 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:52.838733 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:52.838642 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:52.838733 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:52.838720 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:52.838862 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:52.838797 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:52.838934 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:52.838914 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:52.969982 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:52.969940 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="2c398278800bc8e9ff53d51ea6235a0888173e0ebae6c5fa4d4a20a0a43f444e" exitCode=0 May 20 13:50:52.970567 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:52.969994 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"2c398278800bc8e9ff53d51ea6235a0888173e0ebae6c5fa4d4a20a0a43f444e"} May 20 13:50:53.974302 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:53.974265 2575 generic.go:358] "Generic (PLEG): container finished" podID="d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa" containerID="102f6a7cab64d7105e5f6371473bb67b8360fd9263c21baaa36a6a1f205f6c5c" exitCode=0 May 20 13:50:53.974696 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:53.974324 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerDied","Data":"102f6a7cab64d7105e5f6371473bb67b8360fd9263c21baaa36a6a1f205f6c5c"} May 20 13:50:54.838653 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:54.838451 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:54.838950 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:54.838461 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:54.838950 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:54.838472 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:54.838950 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:54.838793 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:54.838950 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:54.838811 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:54.838950 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:54.838891 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:54.978492 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:54.978455 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" event={"ID":"d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa","Type":"ContainerStarted","Data":"c9a996564f280f50b58ac32201a4f7f585139d77c7948a3ad6b32d603e086cdf"} May 20 13:50:55.010475 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:55.010419 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z6nqd" podStartSLOduration=5.842221122 podStartE2EDuration="39.010404857s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.518282786 +0000 UTC m=+3.269095163" lastFinishedPulling="2026-05-20 13:50:51.686466504 +0000 UTC m=+36.437278898" observedRunningTime="2026-05-20 13:50:55.009260119 +0000 UTC m=+39.760072517" watchObservedRunningTime="2026-05-20 13:50:55.010404857 +0000 UTC m=+39.761217253" May 20 13:50:56.838585 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:56.838549 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:56.839016 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:56.838661 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:56.839016 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:56.838669 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:56.839016 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:56.838681 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:56.839016 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:56.838796 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:50:56.839016 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:56.838866 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:58.838834 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:58.838797 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:50:58.839215 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:58.838797 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:50:58.839215 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:58.838901 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:50:58.839215 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:50:58.838804 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:50:58.839215 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:58.838978 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:50:58.839215 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:50:58.839071 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:51:00.838864 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:00.838827 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:00.839331 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:00.838931 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:51:00.839331 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:00.838945 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:00.839331 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:00.838965 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:00.839331 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:00.839064 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:51:00.839331 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:00.839167 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:51:02.838244 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:02.838200 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:02.838244 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:02.838229 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:02.838728 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:02.838229 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:02.838728 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:02.838309 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:51:02.838728 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:02.838395 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:51:02.838728 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:02.838474 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:51:04.839040 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:04.839003 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:04.839396 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:04.839116 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:51:04.839396 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:04.839003 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:04.839396 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:04.839003 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:04.839396 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:04.839223 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:51:04.839396 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:04.839301 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:51:05.619966 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:05.619924 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:05.620155 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:05.620069 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 20 13:51:05.620155 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:05.620133 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret podName:376a9486-73e2-4fe4-b9a1-57419dc0c808 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:37.620114943 +0000 UTC m=+82.370927322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret") pod "global-pull-secret-syncer-kdngv" (UID: "376a9486-73e2-4fe4-b9a1-57419dc0c808") : object "kube-system"/"original-pull-secret" not registered May 20 13:51:06.838641 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:06.838604 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:06.839175 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:06.838604 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:06.839175 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:06.838720 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pjpk" podUID="b8a9e50a-d38e-4123-9a51-fde818feb241" May 20 13:51:06.839175 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:06.838604 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:06.839175 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:06.838840 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlhw8" podUID="c4fb9a1a-e7e7-47f2-af50-76404e0aabfb" May 20 13:51:06.839175 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:06.838918 2575 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdngv" podUID="376a9486-73e2-4fe4-b9a1-57419dc0c808" May 20 13:51:07.085756 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.085726 2575 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeReady" May 20 13:51:07.085925 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.085909 2575 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 20 13:51:07.161505 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.161470 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58f866d876-77fkh"] May 20 13:51:07.180281 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.180250 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.198959 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.198930 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 20 13:51:07.205882 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.205704 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 20 13:51:07.205882 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.205758 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 20 13:51:07.206122 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.205902 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpwm8\"" May 20 13:51:07.232292 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-bound-sa-token\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-installation-pull-secrets\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf842\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-kube-api-access-cf842\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-image-registry-private-configuration\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-tls\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-trusted-ca\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232462 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-certificates\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.232647 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.232526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68932548-56a5-44ca-a92b-2ce59cdb1c1e-ca-trust-extracted\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.240870 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.240833 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 20 13:51:07.268942 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.268913 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58f866d876-77fkh"] May 20 13:51:07.269392 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.269381 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4njk8"] May 20 13:51:07.294641 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.294611 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7rgmg"] May 20 13:51:07.294818 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.294794 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.301287 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.301264 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 20 13:51:07.301422 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.301264 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8jz5j\"" May 20 13:51:07.301422 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.301264 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 20 13:51:07.314131 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.314099 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rgmg"] May 20 13:51:07.314262 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.314138 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4njk8"] May 20 13:51:07.314262 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.314143 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.318239 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.318217 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 20 13:51:07.318379 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.318352 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 20 13:51:07.318525 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.318502 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 20 13:51:07.325356 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.325335 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6l4zd\"" May 20 13:51:07.333094 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333069 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-installation-pull-secrets\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333218 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333102 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cf842\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-kube-api-access-cf842\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333218 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qth\" (UniqueName: \"kubernetes.io/projected/ed6df236-b3f8-4ea9-a6eb-61236e031275-kube-api-access-s2qth\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.333302 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333234 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-image-registry-private-configuration\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333302 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333272 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-tls\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333302 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386a74e3-df07-42cc-adc5-a68db86ebaed-config-volume\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.333419 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386a74e3-df07-42cc-adc5-a68db86ebaed-metrics-tls\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.333419 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386a74e3-df07-42cc-adc5-a68db86ebaed-tmp-dir\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.333419 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333350 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-trusted-ca\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333517 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333425 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-certificates\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333517 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pf7\" (UniqueName: \"kubernetes.io/projected/386a74e3-df07-42cc-adc5-a68db86ebaed-kube-api-access-86pf7\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.333517 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed6df236-b3f8-4ea9-a6eb-61236e031275-cert\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.333517 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333514 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68932548-56a5-44ca-a92b-2ce59cdb1c1e-ca-trust-extracted\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.333695 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.333535 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-bound-sa-token\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.334165 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.334112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68932548-56a5-44ca-a92b-2ce59cdb1c1e-ca-trust-extracted\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.335257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.334506 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-certificates\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.335257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.335039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68932548-56a5-44ca-a92b-2ce59cdb1c1e-trusted-ca\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.338066 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.338035 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-installation-pull-secrets\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.338165 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.338046 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-registry-tls\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.338165 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.338107 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68932548-56a5-44ca-a92b-2ce59cdb1c1e-image-registry-private-configuration\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.348037 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.348008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf842\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-kube-api-access-cf842\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.353819 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.353794 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68932548-56a5-44ca-a92b-2ce59cdb1c1e-bound-sa-token\") pod \"image-registry-58f866d876-77fkh\" (UID: \"68932548-56a5-44ca-a92b-2ce59cdb1c1e\") " pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.397200 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.397166 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-7lq7p"] May 20 13:51:07.423011 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.422933 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:07.424759 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.424712 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-7lq7p"] May 20 13:51:07.430796 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.430768 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7p4p5\"" May 20 13:51:07.431327 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.431306 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" May 20 13:51:07.431480 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.431462 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" May 20 13:51:07.434283 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434265 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386a74e3-df07-42cc-adc5-a68db86ebaed-config-volume\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.434360 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434290 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386a74e3-df07-42cc-adc5-a68db86ebaed-metrics-tls\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.434360 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434307 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386a74e3-df07-42cc-adc5-a68db86ebaed-tmp-dir\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.434360 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvhr\" (UniqueName: \"kubernetes.io/projected/d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5-kube-api-access-whvhr\") pod \"downloads-57bd96b888-7lq7p\" (UID: \"d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5\") " pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:07.434360 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434357 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86pf7\" (UniqueName: \"kubernetes.io/projected/386a74e3-df07-42cc-adc5-a68db86ebaed-kube-api-access-86pf7\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.434571 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434372 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed6df236-b3f8-4ea9-a6eb-61236e031275-cert\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.434571 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434409 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qth\" (UniqueName: \"kubernetes.io/projected/ed6df236-b3f8-4ea9-a6eb-61236e031275-kube-api-access-s2qth\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.434709 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386a74e3-df07-42cc-adc5-a68db86ebaed-tmp-dir\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.434865 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.434848 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386a74e3-df07-42cc-adc5-a68db86ebaed-config-volume\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.436759 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.436724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386a74e3-df07-42cc-adc5-a68db86ebaed-metrics-tls\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.436892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.436876 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed6df236-b3f8-4ea9-a6eb-61236e031275-cert\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.455457 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.455425 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qth\" (UniqueName: \"kubernetes.io/projected/ed6df236-b3f8-4ea9-a6eb-61236e031275-kube-api-access-s2qth\") pod \"ingress-canary-7rgmg\" (UID: \"ed6df236-b3f8-4ea9-a6eb-61236e031275\") " pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.460246 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.460225 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pf7\" (UniqueName: \"kubernetes.io/projected/386a74e3-df07-42cc-adc5-a68db86ebaed-kube-api-access-86pf7\") pod \"dns-default-4njk8\" (UID: \"386a74e3-df07-42cc-adc5-a68db86ebaed\") " pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.489021 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.488981 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:07.534722 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.534692 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-whvhr\" (UniqueName: \"kubernetes.io/projected/d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5-kube-api-access-whvhr\") pod \"downloads-57bd96b888-7lq7p\" (UID: \"d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5\") " pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:07.560342 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.560308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvhr\" (UniqueName: \"kubernetes.io/projected/d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5-kube-api-access-whvhr\") pod \"downloads-57bd96b888-7lq7p\" (UID: \"d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5\") " pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:07.604027 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.603994 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4njk8" May 20 13:51:07.623052 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.622866 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rgmg" May 20 13:51:07.687537 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.684274 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58f866d876-77fkh"] May 20 13:51:07.734253 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.734144 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:07.775783 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.775709 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4njk8"] May 20 13:51:07.779299 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:07.779264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386a74e3_df07_42cc_adc5_a68db86ebaed.slice/crio-0c3ffc69e4d3d435795889e644616d694aacd78f57ac79ed824f926783b588c6 WatchSource:0}: Error finding container 0c3ffc69e4d3d435795889e644616d694aacd78f57ac79ed824f926783b588c6: Status 404 returned error can't find the container with id 0c3ffc69e4d3d435795889e644616d694aacd78f57ac79ed824f926783b588c6 May 20 13:51:07.801670 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.801637 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rgmg"] May 20 13:51:07.805265 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:07.805237 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6df236_b3f8_4ea9_a6eb_61236e031275.slice/crio-6a6abc2114f5592be38b7e87ea273de5f92f3e7c4dab2dfb2815efd198f2e7d8 WatchSource:0}: Error finding container 6a6abc2114f5592be38b7e87ea273de5f92f3e7c4dab2dfb2815efd198f2e7d8: Status 404 returned error can't find the container with id 6a6abc2114f5592be38b7e87ea273de5f92f3e7c4dab2dfb2815efd198f2e7d8 May 20 13:51:07.878366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:07.878333 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-7lq7p"] May 20 13:51:07.881268 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:07.881239 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c5ea91_4d2f_4888_8f1d_83c4d6d830a5.slice/crio-92d6090e8ebbe902f8d735bf6ecb13e56a1fb837eefd214bfd811cd5eeecb9f5 WatchSource:0}: Error finding container 92d6090e8ebbe902f8d735bf6ecb13e56a1fb837eefd214bfd811cd5eeecb9f5: Status 404 returned error can't find the container with id 92d6090e8ebbe902f8d735bf6ecb13e56a1fb837eefd214bfd811cd5eeecb9f5 May 20 13:51:08.004968 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.004863 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4njk8" event={"ID":"386a74e3-df07-42cc-adc5-a68db86ebaed","Type":"ContainerStarted","Data":"0c3ffc69e4d3d435795889e644616d694aacd78f57ac79ed824f926783b588c6"} May 20 13:51:08.006266 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.006243 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58f866d876-77fkh" event={"ID":"68932548-56a5-44ca-a92b-2ce59cdb1c1e","Type":"ContainerStarted","Data":"83c5cdae1a9abfa7a2a7adad82ad250e105689986c9a5016b7d760de10256c0e"} May 20 13:51:08.006266 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.006273 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58f866d876-77fkh" event={"ID":"68932548-56a5-44ca-a92b-2ce59cdb1c1e","Type":"ContainerStarted","Data":"63c3c338b5e539d186c36fedf9439c8a75a0a4de860f06c2e6ff2d4bdbb609e8"} May 20 13:51:08.006457 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.006382 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:08.007344 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.007323 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rgmg" event={"ID":"ed6df236-b3f8-4ea9-a6eb-61236e031275","Type":"ContainerStarted","Data":"6a6abc2114f5592be38b7e87ea273de5f92f3e7c4dab2dfb2815efd198f2e7d8"} May 20 13:51:08.008350 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.008332 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-7lq7p" event={"ID":"d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5","Type":"ContainerStarted","Data":"92d6090e8ebbe902f8d735bf6ecb13e56a1fb837eefd214bfd811cd5eeecb9f5"} May 20 13:51:08.045781 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.045713 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58f866d876-77fkh" podStartSLOduration=2.045697256 podStartE2EDuration="2.045697256s" podCreationTimestamp="2026-05-20 13:51:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:51:08.04449645 +0000 UTC m=+52.795308848" watchObservedRunningTime="2026-05-20 13:51:08.045697256 +0000 UTC m=+52.796509647" May 20 13:51:08.838872 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.838837 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:08.839659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.839173 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:08.839659 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.839189 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:08.842285 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.842082 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 20 13:51:08.843272 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.843251 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 20 13:51:08.843413 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.843397 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 20 13:51:08.843504 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.843490 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qph4s\"" May 20 13:51:08.844280 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.843653 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-68ths\"" May 20 13:51:08.844280 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:08.843251 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 20 13:51:10.121101 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.121074 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl"] May 20 13:51:10.123235 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.123210 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:10.127289 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.127264 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cjlpc\"" May 20 13:51:10.131400 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.131170 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" May 20 13:51:10.162116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.162076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl\" (UID: \"c97bb658-6ebe-40b1-b38c-c975b30f2f4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:10.164056 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.164006 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl"] May 20 13:51:10.262818 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.262766 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl\" (UID: \"c97bb658-6ebe-40b1-b38c-c975b30f2f4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:10.262989 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:10.262923 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found May 20 13:51:10.263046 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:51:10.262995 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates podName:c97bb658-6ebe-40b1-b38c-c975b30f2f4a nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.762974169 +0000 UTC m=+55.513786564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" (UID: "c97bb658-6ebe-40b1-b38c-c975b30f2f4a") : secret "prometheus-operator-admission-webhook-tls" not found May 20 13:51:10.767338 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.767285 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl\" (UID: \"c97bb658-6ebe-40b1-b38c-c975b30f2f4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:10.770109 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:10.770080 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c97bb658-6ebe-40b1-b38c-c975b30f2f4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl\" (UID: \"c97bb658-6ebe-40b1-b38c-c975b30f2f4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:11.017987 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.017898 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4njk8" event={"ID":"386a74e3-df07-42cc-adc5-a68db86ebaed","Type":"ContainerStarted","Data":"5a103cc1ebb9fa3170a2012ea48eb35fd966b6f8a876cdea000eed2b8f648e79"} May 20 13:51:11.017987 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.017944 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4njk8" event={"ID":"386a74e3-df07-42cc-adc5-a68db86ebaed","Type":"ContainerStarted","Data":"ef0525ba73b4a82ab09efeb63f6335d2c5b5240effffd3f49fd3648585014081"} May 20 13:51:11.018203 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.018035 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4njk8" May 20 13:51:11.019828 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.019800 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rgmg" event={"ID":"ed6df236-b3f8-4ea9-a6eb-61236e031275","Type":"ContainerStarted","Data":"7da3eeaaadab2bf084d1e2df272007b50f0af4d87ce0e1e95a68258e1ccfa5f3"} May 20 13:51:11.042640 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.042601 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:11.051209 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.051184 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4njk8_386a74e3-df07-42cc-adc5-a68db86ebaed/kube-rbac-proxy/0.log" May 20 13:51:11.072218 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.072164 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4njk8" podStartSLOduration=1.8284568270000001 podStartE2EDuration="4.072148903s" podCreationTimestamp="2026-05-20 13:51:07 +0000 UTC" firstStartedPulling="2026-05-20 13:51:07.781337528 +0000 UTC m=+52.532149903" lastFinishedPulling="2026-05-20 13:51:10.025029589 +0000 UTC m=+54.775841979" observedRunningTime="2026-05-20 13:51:11.0703455 +0000 UTC m=+55.821157933" watchObservedRunningTime="2026-05-20 13:51:11.072148903 +0000 UTC m=+55.822961301" May 20 13:51:11.116465 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.116390 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7rgmg" podStartSLOduration=1.897656205 podStartE2EDuration="4.116368355s" podCreationTimestamp="2026-05-20 13:51:07 +0000 UTC" firstStartedPulling="2026-05-20 13:51:07.807955069 +0000 UTC m=+52.558767458" lastFinishedPulling="2026-05-20 13:51:10.026667216 +0000 UTC m=+54.777479608" observedRunningTime="2026-05-20 13:51:11.113967365 +0000 UTC m=+55.864779768" watchObservedRunningTime="2026-05-20 13:51:11.116368355 +0000 UTC m=+55.867180756" May 20 13:51:11.226375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:11.226342 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl"] May 20 13:51:11.230150 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:11.230115 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97bb658_6ebe_40b1_b38c_c975b30f2f4a.slice/crio-9d4c2e62c61c469ce31a263f84cc1f039e71a4e272bee8c4934082b539d03fa8 WatchSource:0}: Error finding container 9d4c2e62c61c469ce31a263f84cc1f039e71a4e272bee8c4934082b539d03fa8: Status 404 returned error can't find the container with id 9d4c2e62c61c469ce31a263f84cc1f039e71a4e272bee8c4934082b539d03fa8 May 20 13:51:12.023678 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:12.023633 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" event={"ID":"c97bb658-6ebe-40b1-b38c-c975b30f2f4a","Type":"ContainerStarted","Data":"9d4c2e62c61c469ce31a263f84cc1f039e71a4e272bee8c4934082b539d03fa8"} May 20 13:51:12.038860 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:12.038830 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tzl92_e3c70d7c-144e-473d-9344-f2947744cada/dns-node-resolver/0.log" May 20 13:51:12.229638 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:12.229613 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58f866d876-77fkh_68932548-56a5-44ca-a92b-2ce59cdb1c1e/registry/0.log" May 20 13:51:13.027390 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.027350 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" event={"ID":"c97bb658-6ebe-40b1-b38c-c975b30f2f4a","Type":"ContainerStarted","Data":"c86ed4b1136849a6e83c97c1923862354e4461a8765842c9be79e32452ddac22"} May 20 13:51:13.027590 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.027515 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:13.033717 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.033672 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" May 20 13:51:13.037649 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.037630 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nknqs_25c12a44-9f17-44ff-b915-c31404a6871e/node-ca/0.log" May 20 13:51:13.058226 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.058177 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl" podStartSLOduration=2.070720204 podStartE2EDuration="3.058163069s" podCreationTimestamp="2026-05-20 13:51:10 +0000 UTC" firstStartedPulling="2026-05-20 13:51:11.232703163 +0000 UTC m=+55.983515543" lastFinishedPulling="2026-05-20 13:51:12.220146022 +0000 UTC m=+56.970958408" observedRunningTime="2026-05-20 13:51:13.057795757 +0000 UTC m=+57.808608154" watchObservedRunningTime="2026-05-20 13:51:13.058163069 +0000 UTC m=+57.808975467" May 20 13:51:13.432110 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:13.432072 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7rgmg_ed6df236-b3f8-4ea9-a6eb-61236e031275/serve-healthcheck-canary/0.log" May 20 13:51:14.966677 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:14.966651 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhfql" May 20 13:51:17.179431 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.179397 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hxgd7"] May 20 13:51:17.181795 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.181773 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.201563 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.201531 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 20 13:51:17.202053 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.202035 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 20 13:51:17.202643 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.202578 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 20 13:51:17.202760 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.202644 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dmm4n\"" May 20 13:51:17.202760 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.202692 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 20 13:51:17.221974 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.221940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-crio-socket\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.221974 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.221980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.222214 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.222093 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-data-volume\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.222214 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.222136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.222214 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.222168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9l8\" (UniqueName: \"kubernetes.io/projected/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-api-access-9x9l8\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.322987 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.322953 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxgd7"] May 20 13:51:17.323166 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323029 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323166 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323097 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-data-volume\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323166 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323136 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323304 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323166 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9l8\" (UniqueName: \"kubernetes.io/projected/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-api-access-9x9l8\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323304 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323241 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-crio-socket\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323333 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-crio-socket\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323473 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323445 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-data-volume\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.323879 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.323856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.325736 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.325718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.334709 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.334682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9l8\" (UniqueName: \"kubernetes.io/projected/0c2fbd69-f884-4c3b-99e3-77c25f4af8c7-kube-api-access-9x9l8\") pod \"insights-runtime-extractor-hxgd7\" (UID: \"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7\") " pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:17.492714 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:17.492614 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxgd7" May 20 13:51:19.499769 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.499720 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn"] May 20 13:51:19.502937 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.502911 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.508469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.508420 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" May 20 13:51:19.508773 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.508738 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 20 13:51:19.508877 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.508863 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b6n86\"" May 20 13:51:19.509639 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.509617 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 20 13:51:19.509942 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.509918 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 20 13:51:19.510048 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.509965 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" May 20 13:51:19.540706 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.540659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53128b76-9783-465e-8fe3-fb2b4a93a082-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.540910 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.540767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.540910 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.540795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hvl\" (UniqueName: \"kubernetes.io/projected/53128b76-9783-465e-8fe3-fb2b4a93a082-kube-api-access-c7hvl\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.540910 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.540819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.642247 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.642202 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.642441 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.642253 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hvl\" (UniqueName: \"kubernetes.io/projected/53128b76-9783-465e-8fe3-fb2b4a93a082-kube-api-access-c7hvl\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.642441 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.642286 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.642441 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.642369 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53128b76-9783-465e-8fe3-fb2b4a93a082-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.643188 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.643159 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53128b76-9783-465e-8fe3-fb2b4a93a082-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.645089 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.645067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.645205 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.645092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53128b76-9783-465e-8fe3-fb2b4a93a082-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.698595 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.698558 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn"] May 20 13:51:19.703237 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.703208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hvl\" (UniqueName: \"kubernetes.io/projected/53128b76-9783-465e-8fe3-fb2b4a93a082-kube-api-access-c7hvl\") pod \"openshift-state-metrics-9c959559f-6jqgn\" (UID: \"53128b76-9783-465e-8fe3-fb2b4a93a082\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.715647 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.715617 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wwglq"] May 20 13:51:19.719868 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.718684 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.725368 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.725342 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 20 13:51:19.733447 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.733421 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 20 13:51:19.733631 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.733602 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 20 13:51:19.733919 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.733902 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vchmg\"" May 20 13:51:19.743309 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-sys\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743475 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-textfile\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743475 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-root\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743475 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743608 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-metrics-client-ca\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743608 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-tls\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743608 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z9x\" (UniqueName: \"kubernetes.io/projected/a489e994-139d-4b9d-9af8-5899071bd764-kube-api-access-s5z9x\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.743721 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.743673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-wtmp\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.813703 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.813621 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" May 20 13:51:19.844394 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844354 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-tls\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844566 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844407 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844566 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844445 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z9x\" (UniqueName: \"kubernetes.io/projected/a489e994-139d-4b9d-9af8-5899071bd764-kube-api-access-s5z9x\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844566 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844493 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-wtmp\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844566 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844556 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-sys\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844582 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-textfile\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844619 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-root\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844648 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844671 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-sys\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844680 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-metrics-client-ca\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.844784 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844715 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-wtmp\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.845045 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.844834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a489e994-139d-4b9d-9af8-5899071bd764-root\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.845255 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.845190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.845255 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.845190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a489e994-139d-4b9d-9af8-5899071bd764-metrics-client-ca\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.845440 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.845295 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-textfile\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.847258 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.847237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-tls\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.847355 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.847337 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a489e994-139d-4b9d-9af8-5899071bd764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:19.872880 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:19.872845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z9x\" (UniqueName: \"kubernetes.io/projected/a489e994-139d-4b9d-9af8-5899071bd764-kube-api-access-s5z9x\") pod \"node-exporter-wwglq\" (UID: \"a489e994-139d-4b9d-9af8-5899071bd764\") " pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:20.030368 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:20.030324 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwglq" May 20 13:51:21.026875 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.026844 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4njk8" May 20 13:51:21.661366 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.661320 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:21.661555 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.661377 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:21.664822 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.664792 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 20 13:51:21.665758 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.665708 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 20 13:51:21.674548 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.674522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9e50a-d38e-4123-9a51-fde818feb241-metrics-certs\") pod \"network-metrics-daemon-5pjpk\" (UID: \"b8a9e50a-d38e-4123-9a51-fde818feb241\") " pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:21.676495 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.676466 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 20 13:51:21.684900 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.684872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqsf\" (UniqueName: \"kubernetes.io/projected/c4fb9a1a-e7e7-47f2-af50-76404e0aabfb-kube-api-access-xwqsf\") pod \"network-check-target-dlhw8\" (UID: \"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb\") " pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:21.769272 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.769236 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-68ths\"" May 20 13:51:21.774341 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.774306 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:21.775650 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.775632 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qph4s\"" May 20 13:51:21.781380 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:21.781355 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pjpk" May 20 13:51:23.367229 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.367194 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-94554c77f-9mskt"] May 20 13:51:23.389460 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.389428 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.402066 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.402042 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" May 20 13:51:23.402387 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.402371 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" May 20 13:51:23.402574 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.402555 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xbqkv\"" May 20 13:51:23.403180 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.403160 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" May 20 13:51:23.403323 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.403233 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" May 20 13:51:23.403677 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.403659 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5s9mgoju9r57l\"" May 20 13:51:23.406290 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.406272 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" May 20 13:51:23.455075 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.455043 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94554c77f-9mskt"] May 20 13:51:23.477517 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477715 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477715 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-metrics-client-ca\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477715 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477715 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477925 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477925 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdmf\" (UniqueName: \"kubernetes.io/projected/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-kube-api-access-tvdmf\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.477925 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.477820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-grpc-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.578666 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.578629 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-grpc-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.578887 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.578692 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.578887 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.578729 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.579010 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.578910 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-metrics-client-ca\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.579010 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.578978 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.579156 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.579049 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.579156 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.579089 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.579252 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.579157 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdmf\" (UniqueName: \"kubernetes.io/projected/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-kube-api-access-tvdmf\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.580208 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.580150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-metrics-client-ca\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582020 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.581967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582020 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.582004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-grpc-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582189 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.582162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582284 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.582255 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582619 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.582596 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-tls\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.582619 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.582611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.610088 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.610053 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdmf\" (UniqueName: \"kubernetes.io/projected/cafb9b21-4fe9-4222-b5ec-36e750d3cf37-kube-api-access-tvdmf\") pod \"thanos-querier-94554c77f-9mskt\" (UID: \"cafb9b21-4fe9-4222-b5ec-36e750d3cf37\") " pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.700995 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:23.700910 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:23.841188 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:23.841149 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489e994_139d_4b9d_9af8_5899071bd764.slice/crio-e71ccc636f3b6f6233e8a9e0d0159acd35256a854943ecd3439f0dbd944b1cbb WatchSource:0}: Error finding container e71ccc636f3b6f6233e8a9e0d0159acd35256a854943ecd3439f0dbd944b1cbb: Status 404 returned error can't find the container with id e71ccc636f3b6f6233e8a9e0d0159acd35256a854943ecd3439f0dbd944b1cbb May 20 13:51:24.040837 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.040801 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxgd7"] May 20 13:51:24.044109 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:24.044075 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2fbd69_f884_4c3b_99e3_77c25f4af8c7.slice/crio-446d306ea7a2e7a6323fff8bebe63035158e1d9861a2a3bbde023b834c1ba71d WatchSource:0}: Error finding container 446d306ea7a2e7a6323fff8bebe63035158e1d9861a2a3bbde023b834c1ba71d: Status 404 returned error can't find the container with id 446d306ea7a2e7a6323fff8bebe63035158e1d9861a2a3bbde023b834c1ba71d May 20 13:51:24.054020 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.053989 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-7lq7p" event={"ID":"d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5","Type":"ContainerStarted","Data":"847de3d63effca736e1d76377129e04579f35c984e12eb00238a84cf1b86b705"} May 20 13:51:24.054431 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.054408 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:24.055415 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.055383 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwglq" event={"ID":"a489e994-139d-4b9d-9af8-5899071bd764","Type":"ContainerStarted","Data":"e71ccc636f3b6f6233e8a9e0d0159acd35256a854943ecd3439f0dbd944b1cbb"} May 20 13:51:24.056214 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.056184 2575 patch_prober.go:28] interesting pod/downloads-57bd96b888-7lq7p container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" start-of-body= May 20 13:51:24.056311 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.056243 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-7lq7p" podUID="d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" May 20 13:51:24.056508 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.056486 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxgd7" event={"ID":"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7","Type":"ContainerStarted","Data":"446d306ea7a2e7a6323fff8bebe63035158e1d9861a2a3bbde023b834c1ba71d"} May 20 13:51:24.110355 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.110322 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pjpk"] May 20 13:51:24.112348 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:24.112322 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a9e50a_d38e_4123_9a51_fde818feb241.slice/crio-f23f1be48406e8e0abb8df6f014d0c093c2e80ba46e111d2a50b156c8ec3f0b3 WatchSource:0}: Error finding container f23f1be48406e8e0abb8df6f014d0c093c2e80ba46e111d2a50b156c8ec3f0b3: Status 404 returned error can't find the container with id f23f1be48406e8e0abb8df6f014d0c093c2e80ba46e111d2a50b156c8ec3f0b3 May 20 13:51:24.130367 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.130313 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-7lq7p" podStartSLOduration=1.102751618 podStartE2EDuration="17.130296064s" podCreationTimestamp="2026-05-20 13:51:07 +0000 UTC" firstStartedPulling="2026-05-20 13:51:07.883135451 +0000 UTC m=+52.633947825" lastFinishedPulling="2026-05-20 13:51:23.910679872 +0000 UTC m=+68.661492271" observedRunningTime="2026-05-20 13:51:24.129683328 +0000 UTC m=+68.880495728" watchObservedRunningTime="2026-05-20 13:51:24.130296064 +0000 UTC m=+68.881108460" May 20 13:51:24.271059 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:24.271027 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53128b76_9783_465e_8fe3_fb2b4a93a082.slice/crio-f230cee02543121d0b66e1f947e80e599dea1b60495bf0d9443654806343fb29 WatchSource:0}: Error finding container f230cee02543121d0b66e1f947e80e599dea1b60495bf0d9443654806343fb29: Status 404 returned error can't find the container with id f230cee02543121d0b66e1f947e80e599dea1b60495bf0d9443654806343fb29 May 20 13:51:24.284526 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.284493 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlhw8"] May 20 13:51:24.285288 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:24.285254 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafb9b21_4fe9_4222_b5ec_36e750d3cf37.slice/crio-fe6321243d236609d7de4b959be5457c04bd8b0749b0b10b4a79597f0d6784d8 WatchSource:0}: Error finding container fe6321243d236609d7de4b959be5457c04bd8b0749b0b10b4a79597f0d6784d8: Status 404 returned error can't find the container with id fe6321243d236609d7de4b959be5457c04bd8b0749b0b10b4a79597f0d6784d8 May 20 13:51:24.285613 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.285555 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn"] May 20 13:51:24.287146 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.287119 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94554c77f-9mskt"] May 20 13:51:24.302367 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:24.302254 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fb9a1a_e7e7_47f2_af50_76404e0aabfb.slice/crio-6ebd27b524ddcc0335596f377a6a4998913b161674a46d9526922a8b153b575e WatchSource:0}: Error finding container 6ebd27b524ddcc0335596f377a6a4998913b161674a46d9526922a8b153b575e: Status 404 returned error can't find the container with id 6ebd27b524ddcc0335596f377a6a4998913b161674a46d9526922a8b153b575e May 20 13:51:24.406195 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.403505 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5847d95bf5-t7sjg"] May 20 13:51:24.408164 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.408113 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.420999 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.420687 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" May 20 13:51:24.420999 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.420719 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" May 20 13:51:24.421532 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.421508 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vsjcx\"" May 20 13:51:24.421934 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.421732 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" May 20 13:51:24.424612 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.424485 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8lk1i68qrj9o5\"" May 20 13:51:24.425181 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.424981 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" May 20 13:51:24.450032 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.449986 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5847d95bf5-t7sjg"] May 20 13:51:24.490186 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-client-certs\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66fj\" (UniqueName: \"kubernetes.io/projected/184090c7-5c7a-43bf-a1d6-fbae57f9689c-kube-api-access-r66fj\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-client-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-metrics-server-audit-profiles\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-tls\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.491257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.490456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/184090c7-5c7a-43bf-a1d6-fbae57f9689c-audit-log\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591615 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-client-certs\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591665 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r66fj\" (UniqueName: \"kubernetes.io/projected/184090c7-5c7a-43bf-a1d6-fbae57f9689c-kube-api-access-r66fj\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591707 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-client-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591767 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-metrics-server-audit-profiles\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591813 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-tls\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591882 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/184090c7-5c7a-43bf-a1d6-fbae57f9689c-audit-log\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.591925 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.592778 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.594648 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.593989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/184090c7-5c7a-43bf-a1d6-fbae57f9689c-metrics-server-audit-profiles\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.595892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.595843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/184090c7-5c7a-43bf-a1d6-fbae57f9689c-audit-log\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.597875 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.597766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-client-ca-bundle\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.599399 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.599353 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-tls\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.606612 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.606553 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/184090c7-5c7a-43bf-a1d6-fbae57f9689c-secret-metrics-server-client-certs\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.611119 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.611061 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66fj\" (UniqueName: \"kubernetes.io/projected/184090c7-5c7a-43bf-a1d6-fbae57f9689c-kube-api-access-r66fj\") pod \"metrics-server-5847d95bf5-t7sjg\" (UID: \"184090c7-5c7a-43bf-a1d6-fbae57f9689c\") " pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:24.729068 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:24.728816 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:25.063126 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.063052 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pjpk" event={"ID":"b8a9e50a-d38e-4123-9a51-fde818feb241","Type":"ContainerStarted","Data":"f23f1be48406e8e0abb8df6f014d0c093c2e80ba46e111d2a50b156c8ec3f0b3"} May 20 13:51:25.065346 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.065301 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxgd7" event={"ID":"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7","Type":"ContainerStarted","Data":"7438cf6a160a6ce15477b8158bb59407048c6f997ebfc313d7292e15f95f4204"} May 20 13:51:25.068227 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.068189 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlhw8" event={"ID":"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb","Type":"ContainerStarted","Data":"6ebd27b524ddcc0335596f377a6a4998913b161674a46d9526922a8b153b575e"} May 20 13:51:25.070494 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.070432 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"fe6321243d236609d7de4b959be5457c04bd8b0749b0b10b4a79597f0d6784d8"} May 20 13:51:25.075149 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.075038 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" event={"ID":"53128b76-9783-465e-8fe3-fb2b4a93a082","Type":"ContainerStarted","Data":"7ade54389a31dd8761aa653cef557c51d1b8fc3e1666e498d32dca598a7b934c"} May 20 13:51:25.075149 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.075096 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" event={"ID":"53128b76-9783-465e-8fe3-fb2b4a93a082","Type":"ContainerStarted","Data":"bfb5db6a256fa261cfe8302a5dc2586662aca1ce12752ecf7fc145d7a18ffd31"} May 20 13:51:25.075149 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.075114 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" event={"ID":"53128b76-9783-465e-8fe3-fb2b4a93a082","Type":"ContainerStarted","Data":"f230cee02543121d0b66e1f947e80e599dea1b60495bf0d9443654806343fb29"} May 20 13:51:25.095043 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:25.095008 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-7lq7p" May 20 13:51:26.406294 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.406032 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5847d95bf5-t7sjg"] May 20 13:51:26.627588 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.627550 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:26.659666 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.659630 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.667144 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.666363 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1cj6rshhqcjeu\"" May 20 13:51:26.667144 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.666653 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:26.667144 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.666729 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" May 20 13:51:26.675779 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.673350 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" May 20 13:51:26.678881 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.678847 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" May 20 13:51:26.686944 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.680073 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" May 20 13:51:26.686944 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.680691 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" May 20 13:51:26.686944 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.681123 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" May 20 13:51:26.686944 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.683410 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" May 20 13:51:26.687253 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.687170 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" May 20 13:51:26.701170 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.701042 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" May 20 13:51:26.703866 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.703000 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6rpv\"" May 20 13:51:26.707909 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.704341 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" May 20 13:51:26.707909 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.706420 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.711667 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.710455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711722 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qczs\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.712116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.711986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.728729 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.728697 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813002 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813051 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813099 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813127 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813164 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813197 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813230 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813255 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813279 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qczs\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813310 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813345 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813385 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813409 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813437 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813470 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813531 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.814411 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813564 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.815348 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.813594 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.819393 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.819355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.820493 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.820457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.821617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.821067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.822854 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.822782 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.824032 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.823655 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.824516 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.824491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.825074 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.825049 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.825450 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.825426 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.826168 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.825562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.826168 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.826125 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.826856 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.826761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.828288 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.828062 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.828431 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.828387 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.828726 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.828609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.829138 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.829055 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.829770 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.829714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.833799 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.833678 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:26.842425 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:26.842381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qczs\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs\") pod \"prometheus-k8s-0\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:27.015904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.015624 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:27.132184 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.132026 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxgd7" event={"ID":"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7","Type":"ContainerStarted","Data":"3ce364bc83033cc407ce72994033612fffb1b8e9d310e829023efe6f80f65d62"} May 20 13:51:27.137093 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.137002 2575 generic.go:358] "Generic (PLEG): container finished" podID="a489e994-139d-4b9d-9af8-5899071bd764" containerID="5caa84eb67f2cdc1fd18165d5d063cfddd24e81d0104b727c47b2d0a399b51f5" exitCode=0 May 20 13:51:27.137804 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.137714 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwglq" event={"ID":"a489e994-139d-4b9d-9af8-5899071bd764","Type":"ContainerDied","Data":"5caa84eb67f2cdc1fd18165d5d063cfddd24e81d0104b727c47b2d0a399b51f5"} May 20 13:51:27.140932 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.140902 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" event={"ID":"184090c7-5c7a-43bf-a1d6-fbae57f9689c","Type":"ContainerStarted","Data":"76011d6b1c7cb4fca95ac39b9f3e5be24d69f89049471daf2b0213d5d15f8b4a"} May 20 13:51:27.149442 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.149248 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pjpk" event={"ID":"b8a9e50a-d38e-4123-9a51-fde818feb241","Type":"ContainerStarted","Data":"19967f6a236aa30f289a1dbaa4a38bbcca25ba2676a8e32e075e7b7b944c3104"} May 20 13:51:27.420429 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:27.420354 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:27.425562 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:27.425527 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c6012f_d701_48b5_8791_8aeeaa885650.slice/crio-0a53aad532c519a0d99647a21daa5ee8e9714b3954c78563523883d8bdc2e3ed WatchSource:0}: Error finding container 0a53aad532c519a0d99647a21daa5ee8e9714b3954c78563523883d8bdc2e3ed: Status 404 returned error can't find the container with id 0a53aad532c519a0d99647a21daa5ee8e9714b3954c78563523883d8bdc2e3ed May 20 13:51:28.158651 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.158080 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pjpk" event={"ID":"b8a9e50a-d38e-4123-9a51-fde818feb241","Type":"ContainerStarted","Data":"953c065c4b835c75cbc9b0417c12ae36690494a5dc67cc7d4c8bd95b77472ba2"} May 20 13:51:28.160692 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.160637 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"0a53aad532c519a0d99647a21daa5ee8e9714b3954c78563523883d8bdc2e3ed"} May 20 13:51:28.164738 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.164074 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" event={"ID":"53128b76-9783-465e-8fe3-fb2b4a93a082","Type":"ContainerStarted","Data":"56d4166f2ec105c22935222f289cbbab2d4767e3abb3d531029650e1c5e89d21"} May 20 13:51:28.168787 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.168090 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwglq" event={"ID":"a489e994-139d-4b9d-9af8-5899071bd764","Type":"ContainerStarted","Data":"f2d6067b4f920e9d71c289c2018c944ca14e9bb01e898d266b461a3c7eba2cb5"} May 20 13:51:28.168787 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.168124 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwglq" event={"ID":"a489e994-139d-4b9d-9af8-5899071bd764","Type":"ContainerStarted","Data":"426b8f333025865f208455925c483542a693fcaef9d970256aab556b759a81d8"} May 20 13:51:28.190795 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.190051 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5pjpk" podStartSLOduration=70.083776664 podStartE2EDuration="1m12.190029927s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:51:24.114484427 +0000 UTC m=+68.865296802" lastFinishedPulling="2026-05-20 13:51:26.220737675 +0000 UTC m=+70.971550065" observedRunningTime="2026-05-20 13:51:28.186502194 +0000 UTC m=+72.937314603" watchObservedRunningTime="2026-05-20 13:51:28.190029927 +0000 UTC m=+72.940842325" May 20 13:51:28.226128 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.225029 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6jqgn" podStartSLOduration=6.682617191 podStartE2EDuration="9.225008705s" podCreationTimestamp="2026-05-20 13:51:19 +0000 UTC" firstStartedPulling="2026-05-20 13:51:24.447434703 +0000 UTC m=+69.198247088" lastFinishedPulling="2026-05-20 13:51:26.989826222 +0000 UTC m=+71.740638602" observedRunningTime="2026-05-20 13:51:28.224087644 +0000 UTC m=+72.974900042" watchObservedRunningTime="2026-05-20 13:51:28.225008705 +0000 UTC m=+72.975821103" May 20 13:51:28.266186 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:28.265453 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wwglq" podStartSLOduration=6.893331263 podStartE2EDuration="9.265429564s" podCreationTimestamp="2026-05-20 13:51:19 +0000 UTC" firstStartedPulling="2026-05-20 13:51:23.844433807 +0000 UTC m=+68.595246194" lastFinishedPulling="2026-05-20 13:51:26.216532101 +0000 UTC m=+70.967344495" observedRunningTime="2026-05-20 13:51:28.262270717 +0000 UTC m=+73.013083115" watchObservedRunningTime="2026-05-20 13:51:28.265429564 +0000 UTC m=+73.016241963" May 20 13:51:29.017098 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:29.017066 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58f866d876-77fkh" May 20 13:51:33.190927 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.190887 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" event={"ID":"184090c7-5c7a-43bf-a1d6-fbae57f9689c","Type":"ContainerStarted","Data":"d1ae611e5b498d3e99ccabf2c181da7d8e2757afc867803eb97eeb64362dc019"} May 20 13:51:33.192296 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.192270 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" exitCode=0 May 20 13:51:33.192410 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.192349 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} May 20 13:51:33.194241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.194216 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxgd7" event={"ID":"0c2fbd69-f884-4c3b-99e3-77c25f4af8c7","Type":"ContainerStarted","Data":"c185269658b6fed668f3f01928b472d73bd79c4e540bdfd67dbc10e49454d33f"} May 20 13:51:33.195419 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.195393 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlhw8" event={"ID":"c4fb9a1a-e7e7-47f2-af50-76404e0aabfb","Type":"ContainerStarted","Data":"5946db1a3794355a34e3e2b65f3a786f679b6568fcf77d44c00e78e3c49ecaa4"} May 20 13:51:33.195552 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.195539 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:51:33.197200 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.197181 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"2d106a0681b7799b8ffeefa6fa1d1d46343be3ad5dcc407636fc50d041e6b04e"} May 20 13:51:33.197293 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.197207 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"002e7479348a1241b2b891b21a36dfbe3b20364abce16c2aaddfafb6da28b104"} May 20 13:51:33.197293 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.197222 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"9ae277b42beef914442eb8b81ccd493deebec2b8fde7b5a9076c1092c4851cbf"} May 20 13:51:33.243077 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.243027 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" podStartSLOduration=3.840943369 podStartE2EDuration="9.243012613s" podCreationTimestamp="2026-05-20 13:51:24 +0000 UTC" firstStartedPulling="2026-05-20 13:51:26.780246809 +0000 UTC m=+71.531059184" lastFinishedPulling="2026-05-20 13:51:32.18231605 +0000 UTC m=+76.933128428" observedRunningTime="2026-05-20 13:51:33.241312868 +0000 UTC m=+77.992125267" watchObservedRunningTime="2026-05-20 13:51:33.243012613 +0000 UTC m=+77.993825067" May 20 13:51:33.289681 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:33.289634 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hxgd7" podStartSLOduration=8.209928082 podStartE2EDuration="16.28961872s" podCreationTimestamp="2026-05-20 13:51:17 +0000 UTC" firstStartedPulling="2026-05-20 13:51:24.103021614 +0000 UTC m=+68.853833990" lastFinishedPulling="2026-05-20 13:51:32.182712239 +0000 UTC m=+76.933524628" observedRunningTime="2026-05-20 13:51:33.288273077 +0000 UTC m=+78.039085476" watchObservedRunningTime="2026-05-20 13:51:33.28961872 +0000 UTC m=+78.040431116" May 20 13:51:34.202801 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:34.202738 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"dc7e760b2aa9becc19a06b2cc43abdd17c27a6e55539a6e237b265bb962c5d24"} May 20 13:51:35.209322 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:35.209199 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"8014def71adc88e0971e4860444c5bc648eefc9623f7871f7bace15eecb10600"} May 20 13:51:35.209322 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:35.209241 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" event={"ID":"cafb9b21-4fe9-4222-b5ec-36e750d3cf37","Type":"ContainerStarted","Data":"5109ca164f2453be30aed80f595ef5fab95420796e8da9a75ca26205690b2781"} May 20 13:51:35.209904 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:35.209402 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:35.279916 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:35.279853 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" podStartSLOduration=2.576110191 podStartE2EDuration="12.279821952s" podCreationTimestamp="2026-05-20 13:51:23 +0000 UTC" firstStartedPulling="2026-05-20 13:51:24.288919092 +0000 UTC m=+69.039731473" lastFinishedPulling="2026-05-20 13:51:33.992630847 +0000 UTC m=+78.743443234" observedRunningTime="2026-05-20 13:51:35.265630271 +0000 UTC m=+80.016442697" watchObservedRunningTime="2026-05-20 13:51:35.279821952 +0000 UTC m=+80.030634353" May 20 13:51:35.280160 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:35.280124 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dlhw8" podStartSLOduration=72.404090185 podStartE2EDuration="1m20.280116167s" podCreationTimestamp="2026-05-20 13:50:15 +0000 UTC" firstStartedPulling="2026-05-20 13:51:24.30663909 +0000 UTC m=+69.057451488" lastFinishedPulling="2026-05-20 13:51:32.18266508 +0000 UTC m=+76.933477470" observedRunningTime="2026-05-20 13:51:33.366157832 +0000 UTC m=+78.116970229" watchObservedRunningTime="2026-05-20 13:51:35.280116167 +0000 UTC m=+80.030928566" May 20 13:51:37.217597 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217560 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} May 20 13:51:37.217597 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217596 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} May 20 13:51:37.217597 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217607 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} May 20 13:51:37.218159 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217617 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} May 20 13:51:37.218159 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217624 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} May 20 13:51:37.218159 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.217632 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerStarted","Data":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} May 20 13:51:37.630863 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.630820 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:37.633803 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.633780 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 20 13:51:37.643264 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.643240 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/376a9486-73e2-4fe4-b9a1-57419dc0c808-original-pull-secret\") pod \"global-pull-secret-syncer-kdngv\" (UID: \"376a9486-73e2-4fe4-b9a1-57419dc0c808\") " pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:37.655483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.655451 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdngv" May 20 13:51:37.817174 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.817099 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.092526126 podStartE2EDuration="11.817075856s" podCreationTimestamp="2026-05-20 13:51:26 +0000 UTC" firstStartedPulling="2026-05-20 13:51:27.431678215 +0000 UTC m=+72.182490589" lastFinishedPulling="2026-05-20 13:51:36.156227931 +0000 UTC m=+80.907040319" observedRunningTime="2026-05-20 13:51:37.288069273 +0000 UTC m=+82.038881669" watchObservedRunningTime="2026-05-20 13:51:37.817075856 +0000 UTC m=+82.567888253" May 20 13:51:37.818042 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:37.818019 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kdngv"] May 20 13:51:37.822061 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:51:37.822036 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376a9486_73e2_4fe4_b9a1_57419dc0c808.slice/crio-c9fc7296ac8ca4f521a37da93613194f4e4ae189be50d2829298dc5a2c230ffd WatchSource:0}: Error finding container c9fc7296ac8ca4f521a37da93613194f4e4ae189be50d2829298dc5a2c230ffd: Status 404 returned error can't find the container with id c9fc7296ac8ca4f521a37da93613194f4e4ae189be50d2829298dc5a2c230ffd May 20 13:51:38.221500 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:38.221464 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kdngv" event={"ID":"376a9486-73e2-4fe4-b9a1-57419dc0c808","Type":"ContainerStarted","Data":"c9fc7296ac8ca4f521a37da93613194f4e4ae189be50d2829298dc5a2c230ffd"} May 20 13:51:41.219759 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:41.219713 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-94554c77f-9mskt" May 20 13:51:42.016872 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:42.016837 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:43.245512 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:43.245478 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kdngv" event={"ID":"376a9486-73e2-4fe4-b9a1-57419dc0c808","Type":"ContainerStarted","Data":"5f724b5c53fa65561a016bac9f6e7af7aa9a0e51de3e09d9a0399b03a0497086"} May 20 13:51:43.314381 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:43.314327 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kdngv" podStartSLOduration=65.753137568 podStartE2EDuration="1m10.314310445s" podCreationTimestamp="2026-05-20 13:50:33 +0000 UTC" firstStartedPulling="2026-05-20 13:51:37.824217219 +0000 UTC m=+82.575029598" lastFinishedPulling="2026-05-20 13:51:42.385390087 +0000 UTC m=+87.136202475" observedRunningTime="2026-05-20 13:51:43.313967118 +0000 UTC m=+88.064779525" watchObservedRunningTime="2026-05-20 13:51:43.314310445 +0000 UTC m=+88.065122850" May 20 13:51:44.729161 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:44.729116 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:51:44.729161 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:51:44.729169 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:52:04.205650 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:04.205620 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dlhw8" May 20 13:52:04.734257 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:04.734227 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:52:04.738231 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:04.738203 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5847d95bf5-t7sjg" May 20 13:52:27.016483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:27.016433 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:27.036538 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:27.036504 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:27.383165 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:27.383132 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:48.949113 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949064 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:48.949598 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949548 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="prometheus" containerID="cri-o://0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" gracePeriod=600 May 20 13:52:48.949598 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949556 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" gracePeriod=600 May 20 13:52:48.949717 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949565 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-web" containerID="cri-o://6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" gracePeriod=600 May 20 13:52:48.949717 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949607 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="thanos-sidecar" containerID="cri-o://e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" gracePeriod=600 May 20 13:52:48.949717 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949634 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="config-reloader" containerID="cri-o://6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" gracePeriod=600 May 20 13:52:48.949859 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:48.949693 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy" containerID="cri-o://b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" gracePeriod=600 May 20 13:52:49.200917 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.200854 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.362801 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362737 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qczs\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.362801 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362802 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362826 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362848 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362867 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362889 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362918 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362954 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.362987 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363024 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363011 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363039 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363069 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363108 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363135 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363173 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363267 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363291 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363329 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config\") pod \"c0c6012f-d701-48b5-8791-8aeeaa885650\" (UID: \"c0c6012f-d701-48b5-8791-8aeeaa885650\") " May 20 13:52:49.363894 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.363627 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:49.365429 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.364507 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:49.365429 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.365136 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:49.365429 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.365399 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:49.366283 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.366250 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.366400 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.365959 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:49.366504 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.366469 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.366617 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.366599 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.366780 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.366737 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.367025 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.366989 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:49.367307 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.367279 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs" (OuterVolumeSpecName: "kube-api-access-8qczs") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "kube-api-access-8qczs". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:49.367407 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.367315 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.367407 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.367375 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:49.367559 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.367539 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.367687 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.367671 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out" (OuterVolumeSpecName: "config-out") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:49.368679 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.368664 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config" (OuterVolumeSpecName: "config") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.369038 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.369018 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.378040 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.378014 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config" (OuterVolumeSpecName: "web-config") pod "c0c6012f-d701-48b5-8791-8aeeaa885650" (UID: "c0c6012f-d701-48b5-8791-8aeeaa885650"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:49.433064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433029 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" exitCode=0 May 20 13:52:49.433064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433054 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" exitCode=0 May 20 13:52:49.433064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433064 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" exitCode=0 May 20 13:52:49.433064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433071 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" exitCode=0 May 20 13:52:49.433064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433078 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" exitCode=0 May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433086 2575 generic.go:358] "Generic (PLEG): container finished" podID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" exitCode=0 May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433114 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433148 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433154 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433165 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433186 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433195 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433204 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433215 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0c6012f-d701-48b5-8791-8aeeaa885650","Type":"ContainerDied","Data":"0a53aad532c519a0d99647a21daa5ee8e9714b3954c78563523883d8bdc2e3ed"} May 20 13:52:49.433375 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.433218 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.441209 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.441116 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.448615 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.448595 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.455055 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.455038 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.461643 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.461619 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:49.461866 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.461824 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.464062 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464043 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-metrics-client-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464066 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464078 2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464088 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qczs\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-kube-api-access-8qczs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464097 2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-grpc-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464107 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464116 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464126 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464136 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464136 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464144 2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-kube-rbac-proxy\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464153 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464162 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-config-out\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464170 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-web-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464178 2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464186 2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0c6012f-d701-48b5-8791-8aeeaa885650-secret-metrics-client-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464194 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0c6012f-d701-48b5-8791-8aeeaa885650-tls-assets\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464203 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.464364 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.464210 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0c6012f-d701-48b5-8791-8aeeaa885650-prometheus-k8s-db\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 13:52:49.468358 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.468343 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.475524 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.475501 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:49.476188 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.476176 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.482456 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.482441 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.482694 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.482675 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.482754 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.482702 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.482754 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.482734 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.482981 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.482960 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.483047 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.482990 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.483047 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483013 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.483230 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.483215 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.483267 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483234 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.483267 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483249 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.483445 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.483429 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.483483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483448 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.483483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483460 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.483649 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.483634 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.483722 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483657 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.483722 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483677 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.483904 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.483888 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.483946 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483909 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.483946 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.483922 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.484104 ip-10-0-137-148 kubenswrapper[2575]: E0520 13:52:49.484086 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.484184 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484109 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.484184 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484131 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.484314 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484299 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.484361 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484314 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.484510 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484488 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.484547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484510 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.484715 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484700 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.484788 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484715 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.484954 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484935 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.484999 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.484954 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.485160 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485135 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.485160 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485152 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.485410 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485387 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.485483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485414 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.485634 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485618 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.485671 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485634 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.485846 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485833 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.485901 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.485848 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.486064 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486046 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.486116 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486064 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.486254 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486238 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.486254 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486253 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.486435 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486419 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.486483 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486437 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.486626 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486614 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.486669 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486626 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.486839 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486820 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.486894 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.486839 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.487071 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487048 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.487123 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487071 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.487243 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487228 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.487289 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487244 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.487413 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487398 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.487452 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487412 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.487612 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487594 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.487663 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487612 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.487843 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487824 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.487843 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487842 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.488014 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.487998 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.488014 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488013 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.488212 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488195 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.488263 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488214 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.488402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488386 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.488454 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488402 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.488584 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488569 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.488630 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488584 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.488813 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488794 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.488872 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.488813 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.489060 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489043 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.489101 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489060 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.489235 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489219 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.489288 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489235 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.489436 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489417 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.489490 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489436 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.489657 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489640 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.489696 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489658 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.489887 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489869 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.489887 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.489886 2575 scope.go:117] "RemoveContainer" containerID="dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d" May 20 13:52:49.490080 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490060 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d"} err="failed to get container status \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": rpc error: code = NotFound desc = could not find container \"dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d\": container with ID starting with dd47ce0c3586e704bbabccb9233f25925ec062af3dddb9f0ea6c577b07c31c5d not found: ID does not exist" May 20 13:52:49.490080 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490074 2575 scope.go:117] "RemoveContainer" containerID="b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7" May 20 13:52:49.490246 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490232 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7"} err="failed to get container status \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": rpc error: code = NotFound desc = could not find container \"b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7\": container with ID starting with b98f3ccd7db9aa81e5ca3a469bc8ce89ba9d60c317ed78e45d36a620e0cb95d7 not found: ID does not exist" May 20 13:52:49.490246 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490246 2575 scope.go:117] "RemoveContainer" containerID="6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b" May 20 13:52:49.490421 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490407 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b"} err="failed to get container status \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": rpc error: code = NotFound desc = could not find container \"6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b\": container with ID starting with 6b36707a140cf2469d5b636ce642bdd1fd04371d6f20ee31b51f9571e40cfb3b not found: ID does not exist" May 20 13:52:49.490466 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490421 2575 scope.go:117] "RemoveContainer" containerID="e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df" May 20 13:52:49.490599 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490585 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df"} err="failed to get container status \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": rpc error: code = NotFound desc = could not find container \"e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df\": container with ID starting with e699dda0bf1401f92fac38f09581b6cb87d482b2c57e2736fd3c05cc86a5c2df not found: ID does not exist" May 20 13:52:49.490638 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490599 2575 scope.go:117] "RemoveContainer" containerID="6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193" May 20 13:52:49.490791 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490776 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193"} err="failed to get container status \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": rpc error: code = NotFound desc = could not find container \"6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193\": container with ID starting with 6383b9938ce2aaa8cc61521c9fd9830f804aab1f9d41b201e73837949bcb4193 not found: ID does not exist" May 20 13:52:49.490855 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490791 2575 scope.go:117] "RemoveContainer" containerID="0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613" May 20 13:52:49.491002 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.490982 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613"} err="failed to get container status \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": rpc error: code = NotFound desc = could not find container \"0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613\": container with ID starting with 0363e6813d7eaf9d84aa451b927a57f5d04a860964964651e4a778f8ff527613 not found: ID does not exist" May 20 13:52:49.491074 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.491004 2575 scope.go:117] "RemoveContainer" containerID="a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809" May 20 13:52:49.491215 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.491200 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809"} err="failed to get container status \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": rpc error: code = NotFound desc = could not find container \"a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809\": container with ID starting with a96cb7ef240f9ebcb08c3b9caf264cc7c78a3a046214259fbdbd618ce92d9809 not found: ID does not exist" May 20 13:52:49.523882 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.523852 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:49.524125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524110 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-thanos" May 20 13:52:49.524125 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524125 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-thanos" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524136 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="thanos-sidecar" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524142 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="thanos-sidecar" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524153 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-web" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524158 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-web" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524166 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="config-reloader" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524171 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="config-reloader" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524179 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="init-config-reloader" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524184 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="init-config-reloader" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524190 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524195 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524202 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="prometheus" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524207 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="prometheus" May 20 13:52:49.524241 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524247 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy" May 20 13:52:49.524701 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524255 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-thanos" May 20 13:52:49.524701 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524262 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="kube-rbac-proxy-web" May 20 13:52:49.524701 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524268 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="prometheus" May 20 13:52:49.524701 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524274 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="config-reloader" May 20 13:52:49.524701 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.524282 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" containerName="thanos-sidecar" May 20 13:52:49.527931 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.527913 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.530817 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530789 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" May 20 13:52:49.530941 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530791 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" May 20 13:52:49.530941 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530844 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1cj6rshhqcjeu\"" May 20 13:52:49.530941 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530797 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" May 20 13:52:49.530941 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530829 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" May 20 13:52:49.530941 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.530906 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" May 20 13:52:49.531202 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.531190 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" May 20 13:52:49.531258 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.531219 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" May 20 13:52:49.531304 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.531277 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" May 20 13:52:49.531653 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.531635 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" May 20 13:52:49.531653 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.531641 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" May 20 13:52:49.532392 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.532375 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6rpv\"" May 20 13:52:49.534275 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.534247 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" May 20 13:52:49.537595 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.537579 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" May 20 13:52:49.560822 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.560797 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:49.666114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666114 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666332 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666332 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnmx\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-kube-api-access-ktnmx\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666332 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666332 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666453 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666453 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666453 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666453 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666568 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666568 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666568 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666568 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666679 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666679 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666679 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.666679 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.666645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767614 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767527 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767614 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767562 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767614 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767581 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767713 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767740 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767801 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767841 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.767892 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.767864 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768670 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768705 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768772 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768800 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768834 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768865 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnmx\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-kube-api-access-ktnmx\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768895 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768927 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768959 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.769384 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.768992 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.771031 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.771370 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.771513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.771736 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.771851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.772048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.772168 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772469 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.772420 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.772916 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.772571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.773350 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.773316 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.774956 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.773641 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.774956 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.773968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.774956 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.774883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29bc0749-203a-49a5-bae4-d6c362ad8cf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.775638 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.775616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.777915 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.777893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29bc0749-203a-49a5-bae4-d6c362ad8cf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.784472 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.784450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnmx\" (UniqueName: \"kubernetes.io/projected/29bc0749-203a-49a5-bae4-d6c362ad8cf3-kube-api-access-ktnmx\") pod \"prometheus-k8s-0\" (UID: \"29bc0749-203a-49a5-bae4-d6c362ad8cf3\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.837480 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.837443 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:49.841825 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.841798 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c6012f-d701-48b5-8791-8aeeaa885650" path="/var/lib/kubelet/pods/c0c6012f-d701-48b5-8791-8aeeaa885650/volumes" May 20 13:52:49.982675 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:49.982589 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:49.985286 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:52:49.985259 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bc0749_203a_49a5_bae4_d6c362ad8cf3.slice/crio-af8e845c5e46ee55229a33f322e941619f7b293ca776eef1583005f34be53ed0 WatchSource:0}: Error finding container af8e845c5e46ee55229a33f322e941619f7b293ca776eef1583005f34be53ed0: Status 404 returned error can't find the container with id af8e845c5e46ee55229a33f322e941619f7b293ca776eef1583005f34be53ed0 May 20 13:52:50.438538 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:50.438508 2575 generic.go:358] "Generic (PLEG): container finished" podID="29bc0749-203a-49a5-bae4-d6c362ad8cf3" containerID="3d5a05de863e22eb66e63406bea7ae33c1911df68b464d4b0e4c22eb3aacf3ea" exitCode=0 May 20 13:52:50.438674 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:50.438582 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerDied","Data":"3d5a05de863e22eb66e63406bea7ae33c1911df68b464d4b0e4c22eb3aacf3ea"} May 20 13:52:50.438674 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:50.438602 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"af8e845c5e46ee55229a33f322e941619f7b293ca776eef1583005f34be53ed0"} May 20 13:52:51.445130 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445099 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"010b9a8ca9e479d81216974a9b0d8ceae28298e9b0a562fc817e11a22c8a9b62"} May 20 13:52:51.445130 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445132 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"cabe60f519b074ee42ab22d86d632418397a4e224b7e225fea418a8c85687044"} May 20 13:52:51.445547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445144 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"e86583d4991c0b798220a8ea21db43b77a4c4f7b56c60dbc3bbac378f2dc3c8a"} May 20 13:52:51.445547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445155 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"5d5848941181a3f9a8a8eb6c7dd57a82e17fca3e6bfe18fd40f435133604cea0"} May 20 13:52:51.445547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445163 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"95314d0510612e02331ae24c4645f5da66c7f2875dc9ffc2a1eed2161cb1653f"} May 20 13:52:51.445547 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.445171 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29bc0749-203a-49a5-bae4-d6c362ad8cf3","Type":"ContainerStarted","Data":"1959800ace4d2d9463f4c5d0f413fe15cca36bbc0a576f89c6c0768f99127a90"} May 20 13:52:51.489840 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:51.489630 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.489611085 podStartE2EDuration="2.489611085s" podCreationTimestamp="2026-05-20 13:52:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:51.484428995 +0000 UTC m=+156.235241391" watchObservedRunningTime="2026-05-20 13:52:51.489611085 +0000 UTC m=+156.240423485" May 20 13:52:54.838430 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:52:54.838376 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:49.842028 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:53:49.841995 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:49.854496 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:53:49.854464 2575 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:50.634725 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:53:50.634696 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:55:15.751360 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:55:15.751334 2575 kubelet.go:1628] "Image garbage collection succeeded" May 20 13:58:15.736150 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.736110 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pp88x"] May 20 13:58:15.738551 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.738530 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pp88x" May 20 13:58:15.741288 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.741266 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dln44\"" May 20 13:58:15.741395 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.741354 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 20 13:58:15.742347 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.742330 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 20 13:58:15.742447 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.742412 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" May 20 13:58:15.750696 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.750671 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pp88x"] May 20 13:58:15.841426 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.841394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7h6m\" (UniqueName: \"kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:15.841596 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.841468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:15.942224 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.942191 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:15.942402 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.942259 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g7h6m\" (UniqueName: \"kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:15.942631 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.942610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:15.954273 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:15.954249 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7h6m\" (UniqueName: \"kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m\") pod \"s3-init-pp88x\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " pod="kserve/s3-init-pp88x" May 20 13:58:16.055058 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:16.054971 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pp88x" May 20 13:58:16.181321 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:16.181236 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pp88x"] May 20 13:58:16.183576 ip-10-0-137-148 kubenswrapper[2575]: W0520 13:58:16.183537 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce74e211_4c38_44bc_9cda_96120d8ec124.slice/crio-309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393 WatchSource:0}: Error finding container 309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393: Status 404 returned error can't find the container with id 309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393 May 20 13:58:16.185416 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:16.185399 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 13:58:16.351662 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:16.351575 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerStarted","Data":"309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393"} May 20 13:58:20.364583 ip-10-0-137-148 kubenswrapper[2575]: I0520 13:58:20.364493 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerStarted","Data":"061adad67688908a4abc8bd2ecc51c9f29d9357c798719e27ea16510cd2a534b"} May 20 14:00:15.676726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:00:15.676695 2575 generic.go:358] "Generic (PLEG): container finished" podID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerID="061adad67688908a4abc8bd2ecc51c9f29d9357c798719e27ea16510cd2a534b" exitCode=0 May 20 14:00:15.677139 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:00:15.676774 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerDied","Data":"061adad67688908a4abc8bd2ecc51c9f29d9357c798719e27ea16510cd2a534b"} May 20 14:00:20.182305 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:00:20.182283 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" May 20 14:00:20.693347 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:00:20.693310 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerStarted","Data":"f2dd0fbb9e80aefbbca4c510c0a58372ef48da61b0302cdda3af97e3a65c93ec"} May 20 14:00:20.718319 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:00:20.718265 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-pp88x" podStartSLOduration=1.726016763 podStartE2EDuration="2m5.718251102s" podCreationTimestamp="2026-05-20 13:58:15 +0000 UTC" firstStartedPulling="2026-05-20 13:58:16.185524966 +0000 UTC m=+480.936337341" lastFinishedPulling="2026-05-20 14:00:20.177759288 +0000 UTC m=+604.928571680" observedRunningTime="2026-05-20 14:00:20.717729776 +0000 UTC m=+605.468542186" watchObservedRunningTime="2026-05-20 14:00:20.718251102 +0000 UTC m=+605.469063524" May 20 14:01:25.869880 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:25.869846 2575 generic.go:358] "Generic (PLEG): container finished" podID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerID="f2dd0fbb9e80aefbbca4c510c0a58372ef48da61b0302cdda3af97e3a65c93ec" exitCode=0 May 20 14:01:25.870274 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:25.869907 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerDied","Data":"f2dd0fbb9e80aefbbca4c510c0a58372ef48da61b0302cdda3af97e3a65c93ec"} May 20 14:01:26.999085 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:26.999062 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pp88x" May 20 14:01:27.098284 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.098236 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7h6m\" (UniqueName: \"kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m\") pod \"ce74e211-4c38-44bc-9cda-96120d8ec124\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " May 20 14:01:27.098284 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.098285 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache\") pod \"ce74e211-4c38-44bc-9cda-96120d8ec124\" (UID: \"ce74e211-4c38-44bc-9cda-96120d8ec124\") " May 20 14:01:27.100380 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.100350 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m" (OuterVolumeSpecName: "kube-api-access-g7h6m") pod "ce74e211-4c38-44bc-9cda-96120d8ec124" (UID: "ce74e211-4c38-44bc-9cda-96120d8ec124"). InnerVolumeSpecName "kube-api-access-g7h6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:01:27.120200 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.120170 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce74e211-4c38-44bc-9cda-96120d8ec124" (UID: "ce74e211-4c38-44bc-9cda-96120d8ec124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:01:27.199389 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.199308 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7h6m\" (UniqueName: \"kubernetes.io/projected/ce74e211-4c38-44bc-9cda-96120d8ec124-kube-api-access-g7h6m\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:01:27.199389 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.199337 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce74e211-4c38-44bc-9cda-96120d8ec124-model-cache\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:01:27.877150 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.877118 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pp88x" event={"ID":"ce74e211-4c38-44bc-9cda-96120d8ec124","Type":"ContainerDied","Data":"309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393"} May 20 14:01:27.877150 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.877141 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pp88x" May 20 14:01:27.877314 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:27.877156 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309a6a70e8ced42a86dd3f06d75100d9dc05f5fe0ac8bf5500f8850607c05393" May 20 14:01:38.143024 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.142990 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:01:38.143522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.143389 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerName="download-hf-model" May 20 14:01:38.143522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.143406 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerName="download-hf-model" May 20 14:01:38.143522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.143443 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerName="s3-init" May 20 14:01:38.143522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.143451 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerName="s3-init" May 20 14:01:38.143522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.143514 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce74e211-4c38-44bc-9cda-96120d8ec124" containerName="s3-init" May 20 14:01:38.146778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.146760 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.149921 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.149898 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" May 20 14:01:38.150521 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.150505 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" May 20 14:01:38.150827 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.150811 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" May 20 14:01:38.150904 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.150821 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-k2v42\"" May 20 14:01:38.150904 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.150844 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" May 20 14:01:38.151080 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.151066 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 20 14:01:38.151158 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.151142 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 20 14:01:38.158387 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.158368 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:01:38.183947 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.183922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.184052 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.183965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.184052 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.184027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.184052 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.184048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.184166 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.184112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86hh\" (UniqueName: \"kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284653 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.284612 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.284671 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.284731 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.284784 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.284829 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n86hh\" (UniqueName: \"kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.284869 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:01:38.284852 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found May 20 14:01:38.285129 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:01:38.284928 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls podName:efd1378d-8fc2-43b0-9a18-4cce7fac91f9 nodeName:}" failed. No retries permitted until 2026-05-20 14:01:38.784912446 +0000 UTC m=+683.535724821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls") pod "isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found May 20 14:01:38.285200 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.285181 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.285471 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.285450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.285643 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.285626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.298687 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.298655 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:01:38.301982 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.301964 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.304633 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.304616 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69925-predictor-serving-cert\"" May 20 14:01:38.305429 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.305409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86hh\" (UniqueName: \"kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.306052 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.306031 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69925-kube-rbac-proxy-sar-config\"" May 20 14:01:38.321478 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.321458 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:01:38.386342 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.386310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.386466 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.386386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.386466 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.386451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mn9\" (UniqueName: \"kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.487268 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.487187 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mn9\" (UniqueName: \"kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.487268 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.487247 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.487467 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.487280 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.488025 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.487997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.489640 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.489620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.510115 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.510091 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mn9\" (UniqueName: \"kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9\") pod \"success-200-isvc-69925-predictor-697c88cf8-dk2sh\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.617183 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.617156 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:38.699285 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.699249 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:01:38.703977 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.703956 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.706560 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.706539 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" May 20 14:01:38.706659 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.706539 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" May 20 14:01:38.715848 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.715825 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:01:38.779573 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.779508 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:01:38.782588 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:01:38.782564 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacdadf1_fcb3_46b2_b379_1cf4e772e64f.slice/crio-ce71d71165a6b1b01f4215b034b14e9b72c231ac397fa3a6a27212e5a281e3c9 WatchSource:0}: Error finding container ce71d71165a6b1b01f4215b034b14e9b72c231ac397fa3a6a27212e5a281e3c9: Status 404 returned error can't find the container with id ce71d71165a6b1b01f4215b034b14e9b72c231ac397fa3a6a27212e5a281e3c9 May 20 14:01:38.790537 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790513 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.790626 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f76m\" (UniqueName: \"kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.790626 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.790626 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.790980 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.790980 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.790659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.792810 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.792793 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:38.891793 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.891762 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7f76m\" (UniqueName: \"kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.891948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.891801 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.891948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.891833 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.891948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.891862 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.891948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.891892 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.892297 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.892266 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.892577 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.892555 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.892643 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.892574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.894132 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.894114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.902064 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.902046 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f76m\" (UniqueName: \"kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m\") pod \"isvc-xgboost-graph-predictor-669d8d6456-xhv8r\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:38.908413 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:38.908391 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerStarted","Data":"ce71d71165a6b1b01f4215b034b14e9b72c231ac397fa3a6a27212e5a281e3c9"} May 20 14:01:39.016782 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.016734 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:01:39.056604 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.056522 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:39.155504 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.155473 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:01:39.157514 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:01:39.157488 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd098d1_0850_4e94_8fba_0197f4b1f733.slice/crio-ab98a3fd87e43b8c53292c328b2a923dfc4516f250ea098e582e1ada44ae7124 WatchSource:0}: Error finding container ab98a3fd87e43b8c53292c328b2a923dfc4516f250ea098e582e1ada44ae7124: Status 404 returned error can't find the container with id ab98a3fd87e43b8c53292c328b2a923dfc4516f250ea098e582e1ada44ae7124 May 20 14:01:39.203758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.203712 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:01:39.206568 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:01:39.206548 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd1378d_8fc2_43b0_9a18_4cce7fac91f9.slice/crio-ff83eb3465d908a279c17f778b570961f68d288b172171688cc7a80fd790b959 WatchSource:0}: Error finding container ff83eb3465d908a279c17f778b570961f68d288b172171688cc7a80fd790b959: Status 404 returned error can't find the container with id ff83eb3465d908a279c17f778b570961f68d288b172171688cc7a80fd790b959 May 20 14:01:39.505358 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.505322 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:01:39.509341 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.509316 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.518700 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.517945 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" May 20 14:01:39.518700 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.518223 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" May 20 14:01:39.535886 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.535833 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:01:39.598292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.597982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv98h\" (UniqueName: \"kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.598292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.598047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.598292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.598078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.598292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.598131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.598292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.598202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.699595 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.699671 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dv98h\" (UniqueName: \"kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.699723 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.699784 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.699836 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.701778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.701449 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.702896 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.702771 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.703533 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.703473 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.703856 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.703807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.711994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.711953 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv98h\" (UniqueName: \"kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h\") pod \"isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.823835 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.823658 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:39.933687 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.933596 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerStarted","Data":"8ddb55715fc3e7f4607d76f56d265c29779c84df49685268f02b0261b19d920d"} May 20 14:01:39.933687 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.933645 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerStarted","Data":"ab98a3fd87e43b8c53292c328b2a923dfc4516f250ea098e582e1ada44ae7124"} May 20 14:01:39.943769 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.943717 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerStarted","Data":"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd"} May 20 14:01:39.943994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:39.943967 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerStarted","Data":"ff83eb3465d908a279c17f778b570961f68d288b172171688cc7a80fd790b959"} May 20 14:01:40.622864 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:40.622790 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:01:40.626681 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:01:40.626327 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c4ca2_5c86_491f_8694_a7904ffd2bc8.slice/crio-5abb033b51ffb536003a5d2f91432f8adc941bda4e9cc311a1307e758de63641 WatchSource:0}: Error finding container 5abb033b51ffb536003a5d2f91432f8adc941bda4e9cc311a1307e758de63641: Status 404 returned error can't find the container with id 5abb033b51ffb536003a5d2f91432f8adc941bda4e9cc311a1307e758de63641 May 20 14:01:40.949786 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:40.949658 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerStarted","Data":"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a"} May 20 14:01:40.949786 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:40.949737 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerStarted","Data":"5abb033b51ffb536003a5d2f91432f8adc941bda4e9cc311a1307e758de63641"} May 20 14:01:43.961594 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:43.961564 2575 generic.go:358] "Generic (PLEG): container finished" podID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerID="8ddb55715fc3e7f4607d76f56d265c29779c84df49685268f02b0261b19d920d" exitCode=0 May 20 14:01:43.962118 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:43.961654 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerDied","Data":"8ddb55715fc3e7f4607d76f56d265c29779c84df49685268f02b0261b19d920d"} May 20 14:01:43.963265 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:43.963167 2575 generic.go:358] "Generic (PLEG): container finished" podID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerID="5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd" exitCode=0 May 20 14:01:43.963265 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:43.963208 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerDied","Data":"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd"} May 20 14:01:48.981402 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:48.981175 2575 generic.go:358] "Generic (PLEG): container finished" podID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerID="d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a" exitCode=0 May 20 14:01:48.981402 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:48.981276 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerDied","Data":"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a"} May 20 14:01:51.997410 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:51.997360 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerStarted","Data":"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2"} May 20 14:01:56.014547 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:56.014500 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerStarted","Data":"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe"} May 20 14:01:56.016231 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:56.016202 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerStarted","Data":"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89"} May 20 14:01:59.029154 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.029111 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerStarted","Data":"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2"} May 20 14:01:59.029574 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.029254 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:01:59.030523 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.030498 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerStarted","Data":"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64"} May 20 14:01:59.030639 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.030613 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:01:59.032177 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.032152 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerStarted","Data":"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572"} May 20 14:01:59.032352 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.032334 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:59.032418 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.032362 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:01:59.034153 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.034128 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:01:59.060792 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.060735 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podStartSLOduration=6.487056794 podStartE2EDuration="21.060719023s" podCreationTimestamp="2026-05-20 14:01:38 +0000 UTC" firstStartedPulling="2026-05-20 14:01:43.964266102 +0000 UTC m=+688.715078477" lastFinishedPulling="2026-05-20 14:01:58.537928317 +0000 UTC m=+703.288740706" observedRunningTime="2026-05-20 14:01:59.05783663 +0000 UTC m=+703.808649027" watchObservedRunningTime="2026-05-20 14:01:59.060719023 +0000 UTC m=+703.811531420" May 20 14:01:59.095665 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.095621 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podStartSLOduration=1.371885199 podStartE2EDuration="21.09560713s" podCreationTimestamp="2026-05-20 14:01:38 +0000 UTC" firstStartedPulling="2026-05-20 14:01:38.784241338 +0000 UTC m=+683.535053713" lastFinishedPulling="2026-05-20 14:01:58.507963263 +0000 UTC m=+703.258775644" observedRunningTime="2026-05-20 14:01:59.094416904 +0000 UTC m=+703.845229303" watchObservedRunningTime="2026-05-20 14:01:59.09560713 +0000 UTC m=+703.846419526" May 20 14:01:59.123138 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:01:59.123027 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podStartSLOduration=10.579853382 podStartE2EDuration="20.123009779s" podCreationTimestamp="2026-05-20 14:01:39 +0000 UTC" firstStartedPulling="2026-05-20 14:01:48.982654941 +0000 UTC m=+693.733467317" lastFinishedPulling="2026-05-20 14:01:58.525811324 +0000 UTC m=+703.276623714" observedRunningTime="2026-05-20 14:01:59.120394683 +0000 UTC m=+703.871207092" watchObservedRunningTime="2026-05-20 14:01:59.123009779 +0000 UTC m=+703.873822178" May 20 14:02:00.035963 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:00.035921 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:00.036425 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:00.036342 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:02:00.036425 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:00.036385 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:02:00.037533 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:00.037505 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:00.037654 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:00.037527 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:01.039050 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:01.039010 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:01.039483 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:01.039009 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:05.041571 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:05.041527 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:02:05.042133 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:05.042108 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:06.043864 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:06.043830 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:02:06.044651 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:06.044630 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:02:06.044800 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:06.044691 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:06.045259 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:06.045215 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:12.083484 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:12.083446 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerStarted","Data":"bc4ecd50c8569165b6eeb85d3a108b18f912a0de68e8774c7396e1f796328d07"} May 20 14:02:12.083903 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:12.083491 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerStarted","Data":"4f47f767ba0b4f1ccb6dc9bbecbccf25787f5af0339996747acf97bb2365e812"} May 20 14:02:12.083903 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:12.083776 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:02:13.087067 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:13.087029 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:02:13.088299 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:13.088276 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:14.090232 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:14.090198 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:15.042160 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:15.042123 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:16.045512 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:16.045458 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:16.046016 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:16.045458 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:19.095070 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:19.095038 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:02:19.095689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:19.095662 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:19.126643 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:19.126598 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podStartSLOduration=13.954104559 podStartE2EDuration="41.126585015s" podCreationTimestamp="2026-05-20 14:01:38 +0000 UTC" firstStartedPulling="2026-05-20 14:01:43.963222096 +0000 UTC m=+688.714034472" lastFinishedPulling="2026-05-20 14:02:11.135702552 +0000 UTC m=+715.886514928" observedRunningTime="2026-05-20 14:02:12.125577463 +0000 UTC m=+716.876389860" watchObservedRunningTime="2026-05-20 14:02:19.126585015 +0000 UTC m=+723.877397413" May 20 14:02:25.042762 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:25.042696 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:26.045383 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:26.045337 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:26.045911 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:26.045350 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:29.095833 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:29.095792 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:35.042653 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:35.042617 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:36.044901 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:36.044854 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:36.045444 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:36.045414 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:39.096554 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:39.096515 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:45.042292 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:45.042248 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:46.045063 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:46.045015 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:46.045530 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:46.045205 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" May 20 14:02:49.095857 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:49.095817 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:02:55.042793 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:55.042735 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:02:56.045693 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:56.045650 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:02:56.046379 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:56.046349 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:02:59.095711 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:02:59.095663 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:03:05.042216 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:05.042177 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:03:06.045053 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:06.045004 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:03:06.839249 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:06.839204 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" May 20 14:03:09.095794 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:09.095735 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:03:16.045646 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:16.045618 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:03:16.840698 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:16.840667 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:03:18.514908 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.514877 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:03:18.515276 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.515153 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" containerID="cri-o://acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2" gracePeriod=30 May 20 14:03:18.515276 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.515194 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kube-rbac-proxy" containerID="cri-o://f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64" gracePeriod=30 May 20 14:03:18.580924 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.580890 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:03:18.583411 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.583395 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.587366 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.587337 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-83116-predictor-serving-cert\"" May 20 14:03:18.587503 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.587375 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-83116-kube-rbac-proxy-sar-config\"" May 20 14:03:18.597428 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.597392 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:03:18.666038 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.666000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.666199 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.666067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.666199 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.666091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgxx\" (UniqueName: \"kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.767543 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.767445 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.767543 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.767517 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.767543 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.767543 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgxx\" (UniqueName: \"kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.767886 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:18.767680 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-83116-predictor-serving-cert: secret "success-200-isvc-83116-predictor-serving-cert" not found May 20 14:03:18.767886 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:18.767792 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls podName:5b59113e-1550-4804-b847-d4e3d67f7931 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:19.267768404 +0000 UTC m=+784.018580793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls") pod "success-200-isvc-83116-predictor-59d5fb4b86-m27ms" (UID: "5b59113e-1550-4804-b847-d4e3d67f7931") : secret "success-200-isvc-83116-predictor-serving-cert" not found May 20 14:03:18.768085 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.768065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:18.782365 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:18.782334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgxx\" (UniqueName: \"kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:19.096584 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.096503 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:03:19.271583 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.271545 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:19.273881 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.273862 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") pod \"success-200-isvc-83116-predictor-59d5fb4b86-m27ms\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:19.289522 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.289493 2575 generic.go:358] "Generic (PLEG): container finished" podID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerID="f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64" exitCode=2 May 20 14:03:19.289637 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.289530 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerDied","Data":"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64"} May 20 14:03:19.494472 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.494431 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:19.625482 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.625375 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:03:19.627994 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:03:19.627963 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b59113e_1550_4804_b847_d4e3d67f7931.slice/crio-2437c32337df9a12da48fa7ac005f3ff85839a3e0c627e401f960076688ffa67 WatchSource:0}: Error finding container 2437c32337df9a12da48fa7ac005f3ff85839a3e0c627e401f960076688ffa67: Status 404 returned error can't find the container with id 2437c32337df9a12da48fa7ac005f3ff85839a3e0c627e401f960076688ffa67 May 20 14:03:19.629664 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:19.629648 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:03:20.294243 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:20.294204 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerStarted","Data":"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371"} May 20 14:03:20.294243 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:20.294242 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerStarted","Data":"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3"} May 20 14:03:20.294243 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:20.294253 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerStarted","Data":"2437c32337df9a12da48fa7ac005f3ff85839a3e0c627e401f960076688ffa67"} May 20 14:03:20.294550 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:20.294327 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:20.316916 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:20.316871 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podStartSLOduration=2.316857144 podStartE2EDuration="2.316857144s" podCreationTimestamp="2026-05-20 14:03:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:20.315085195 +0000 UTC m=+785.065897592" watchObservedRunningTime="2026-05-20 14:03:20.316857144 +0000 UTC m=+785.067669541" May 20 14:03:21.039574 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.039522 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused" May 20 14:03:21.297237 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.297143 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:21.298330 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.298306 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:21.663011 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.662983 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:03:21.693412 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.693383 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls\") pod \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " May 20 14:03:21.693565 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.693437 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mn9\" (UniqueName: \"kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9\") pod \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " May 20 14:03:21.693565 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.693517 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config\") pod \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\" (UID: \"eacdadf1-fcb3-46b2-b379-1cf4e772e64f\") " May 20 14:03:21.693932 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.693898 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-69925-kube-rbac-proxy-sar-config") pod "eacdadf1-fcb3-46b2-b379-1cf4e772e64f" (UID: "eacdadf1-fcb3-46b2-b379-1cf4e772e64f"). InnerVolumeSpecName "success-200-isvc-69925-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:21.695872 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.695833 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eacdadf1-fcb3-46b2-b379-1cf4e772e64f" (UID: "eacdadf1-fcb3-46b2-b379-1cf4e772e64f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:21.695984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.695881 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9" (OuterVolumeSpecName: "kube-api-access-g6mn9") pod "eacdadf1-fcb3-46b2-b379-1cf4e772e64f" (UID: "eacdadf1-fcb3-46b2-b379-1cf4e772e64f"). InnerVolumeSpecName "kube-api-access-g6mn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:21.794588 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.794551 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-69925-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-success-200-isvc-69925-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:21.794588 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.794581 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:21.794588 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:21.794592 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mn9\" (UniqueName: \"kubernetes.io/projected/eacdadf1-fcb3-46b2-b379-1cf4e772e64f-kube-api-access-g6mn9\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:22.300568 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.300535 2575 generic.go:358] "Generic (PLEG): container finished" podID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerID="acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2" exitCode=0 May 20 14:03:22.301045 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.300610 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" May 20 14:03:22.301045 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.300615 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerDied","Data":"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2"} May 20 14:03:22.301045 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.300660 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh" event={"ID":"eacdadf1-fcb3-46b2-b379-1cf4e772e64f","Type":"ContainerDied","Data":"ce71d71165a6b1b01f4215b034b14e9b72c231ac397fa3a6a27212e5a281e3c9"} May 20 14:03:22.301045 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.300684 2575 scope.go:117] "RemoveContainer" containerID="f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64" May 20 14:03:22.301234 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.301111 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:22.308388 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.308371 2575 scope.go:117] "RemoveContainer" containerID="acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2" May 20 14:03:22.315495 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.315472 2575 scope.go:117] "RemoveContainer" containerID="f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64" May 20 14:03:22.315775 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:22.315738 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64\": container with ID starting with f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64 not found: ID does not exist" containerID="f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64" May 20 14:03:22.315827 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.315786 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64"} err="failed to get container status \"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64\": rpc error: code = NotFound desc = could not find container \"f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64\": container with ID starting with f964658318ade450566446d8a39b3ce26fc0f2cf7c396a7fe9c0d79eb288dc64 not found: ID does not exist" May 20 14:03:22.315827 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.315804 2575 scope.go:117] "RemoveContainer" containerID="acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2" May 20 14:03:22.316032 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:22.316014 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2\": container with ID starting with acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2 not found: ID does not exist" containerID="acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2" May 20 14:03:22.316073 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.316037 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2"} err="failed to get container status \"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2\": rpc error: code = NotFound desc = could not find container \"acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2\": container with ID starting with acbdca1eb79cee8b63b63aeeec3019c6a2b7047f42ee0b5cd38d51d0e200f6b2 not found: ID does not exist" May 20 14:03:22.329195 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.329173 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:03:22.340297 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:22.340273 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69925-predictor-697c88cf8-dk2sh"] May 20 14:03:23.842382 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:23.842348 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" path="/var/lib/kubelet/pods/eacdadf1-fcb3-46b2-b379-1cf4e772e64f/volumes" May 20 14:03:27.306416 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:27.306386 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:03:27.307027 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:27.306936 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:37.307389 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:37.307310 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:47.307811 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:47.307769 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:48.564657 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.564615 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:03:48.565175 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.564949 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" containerID="cri-o://86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe" gracePeriod=30 May 20 14:03:48.565175 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.565005 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" containerID="cri-o://e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2" gracePeriod=30 May 20 14:03:48.585951 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.585916 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:03:48.586343 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586329 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kube-rbac-proxy" May 20 14:03:48.586384 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586345 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kube-rbac-proxy" May 20 14:03:48.586384 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586367 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" May 20 14:03:48.586384 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586373 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" May 20 14:03:48.589455 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586825 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kserve-container" May 20 14:03:48.589455 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.586908 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="eacdadf1-fcb3-46b2-b379-1cf4e772e64f" containerName="kube-rbac-proxy" May 20 14:03:48.590939 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.590918 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.595163 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.595140 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\"" May 20 14:03:48.595271 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.595147 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cdd1d-predictor-serving-cert\"" May 20 14:03:48.600121 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.600098 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:03:48.650506 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.650470 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:03:48.650964 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.650910 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" containerID="cri-o://4f47f767ba0b4f1ccb6dc9bbecbccf25787f5af0339996747acf97bb2365e812" gracePeriod=30 May 20 14:03:48.650964 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.650945 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kube-rbac-proxy" containerID="cri-o://bc4ecd50c8569165b6eeb85d3a108b18f912a0de68e8774c7396e1f796328d07" gracePeriod=30 May 20 14:03:48.720484 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.720449 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:03:48.720804 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.720774 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" containerID="cri-o://875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89" gracePeriod=30 May 20 14:03:48.720951 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.720814 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kube-rbac-proxy" containerID="cri-o://d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572" gracePeriod=30 May 20 14:03:48.739988 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.739037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.739988 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.739112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swxh\" (UniqueName: \"kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.739988 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.739154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.840317 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.840272 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.840490 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.840358 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.840490 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.840446 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4swxh\" (UniqueName: \"kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.840613 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:48.840525 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-serving-cert: secret "success-200-isvc-cdd1d-predictor-serving-cert" not found May 20 14:03:48.840613 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:48.840605 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls podName:c89fd641-6384-45fc-9a9b-eb65d310c2e3 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:49.340587774 +0000 UTC m=+814.091400158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls") pod "success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" (UID: "c89fd641-6384-45fc-9a9b-eb65d310c2e3") : secret "success-200-isvc-cdd1d-predictor-serving-cert" not found May 20 14:03:48.840976 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.840951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:48.851804 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:48.851735 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swxh\" (UniqueName: \"kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:49.090661 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.090560 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused" May 20 14:03:49.095964 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.095935 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused" May 20 14:03:49.346267 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.346163 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:49.348659 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.348613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") pod \"success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:49.381599 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.381569 2575 generic.go:358] "Generic (PLEG): container finished" podID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerID="e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2" exitCode=2 May 20 14:03:49.381802 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.381646 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerDied","Data":"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2"} May 20 14:03:49.383303 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.383284 2575 generic.go:358] "Generic (PLEG): container finished" podID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerID="d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572" exitCode=2 May 20 14:03:49.383396 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.383355 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerDied","Data":"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572"} May 20 14:03:49.384978 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.384958 2575 generic.go:358] "Generic (PLEG): container finished" podID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerID="bc4ecd50c8569165b6eeb85d3a108b18f912a0de68e8774c7396e1f796328d07" exitCode=2 May 20 14:03:49.385073 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.385006 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerDied","Data":"bc4ecd50c8569165b6eeb85d3a108b18f912a0de68e8774c7396e1f796328d07"} May 20 14:03:49.504390 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.504355 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:49.634889 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:49.634853 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:03:49.638001 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:03:49.637952 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89fd641_6384_45fc_9a9b_eb65d310c2e3.slice/crio-087946862fb563192040f866a72a75888eca6c60365072e40f149598080cb1e9 WatchSource:0}: Error finding container 087946862fb563192040f866a72a75888eca6c60365072e40f149598080cb1e9: Status 404 returned error can't find the container with id 087946862fb563192040f866a72a75888eca6c60365072e40f149598080cb1e9 May 20 14:03:50.037111 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.037018 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused" May 20 14:03:50.389703 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.389665 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerStarted","Data":"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85"} May 20 14:03:50.389703 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.389707 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerStarted","Data":"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3"} May 20 14:03:50.389932 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.389719 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerStarted","Data":"087946862fb563192040f866a72a75888eca6c60365072e40f149598080cb1e9"} May 20 14:03:50.389932 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.389804 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:50.409941 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:50.409888 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podStartSLOduration=2.409872916 podStartE2EDuration="2.409872916s" podCreationTimestamp="2026-05-20 14:03:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:50.409158426 +0000 UTC m=+815.159970827" watchObservedRunningTime="2026-05-20 14:03:50.409872916 +0000 UTC m=+815.160685308" May 20 14:03:51.040007 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:51.039968 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused" May 20 14:03:51.393141 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:51.393103 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:51.394490 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:51.394461 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:03:52.395865 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:52.395825 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:03:53.379340 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.379316 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:03:53.400174 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.400147 2575 generic.go:358] "Generic (PLEG): container finished" podID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerID="875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89" exitCode=0 May 20 14:03:53.400546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.400204 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerDied","Data":"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89"} May 20 14:03:53.400546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.400234 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" event={"ID":"d35c4ca2-5c86-491f-8694-a7904ffd2bc8","Type":"ContainerDied","Data":"5abb033b51ffb536003a5d2f91432f8adc941bda4e9cc311a1307e758de63641"} May 20 14:03:53.400546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.400251 2575 scope.go:117] "RemoveContainer" containerID="d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572" May 20 14:03:53.400546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.400293 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg" May 20 14:03:53.402710 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.402457 2575 generic.go:358] "Generic (PLEG): container finished" podID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerID="4f47f767ba0b4f1ccb6dc9bbecbccf25787f5af0339996747acf97bb2365e812" exitCode=0 May 20 14:03:53.402710 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.402483 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerDied","Data":"4f47f767ba0b4f1ccb6dc9bbecbccf25787f5af0339996747acf97bb2365e812"} May 20 14:03:53.410738 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.410697 2575 scope.go:117] "RemoveContainer" containerID="875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89" May 20 14:03:53.411710 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.411685 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:03:53.418557 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.418539 2575 scope.go:117] "RemoveContainer" containerID="d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a" May 20 14:03:53.425670 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.425650 2575 scope.go:117] "RemoveContainer" containerID="d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572" May 20 14:03:53.425943 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:53.425923 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572\": container with ID starting with d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572 not found: ID does not exist" containerID="d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572" May 20 14:03:53.425993 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.425957 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572"} err="failed to get container status \"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572\": rpc error: code = NotFound desc = could not find container \"d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572\": container with ID starting with d3fc861ca5b840ce05ad9983e8686fb5780e857b96a92436d22507a86083f572 not found: ID does not exist" May 20 14:03:53.425993 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.425978 2575 scope.go:117] "RemoveContainer" containerID="875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89" May 20 14:03:53.426189 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:53.426172 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89\": container with ID starting with 875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89 not found: ID does not exist" containerID="875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89" May 20 14:03:53.426225 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.426195 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89"} err="failed to get container status \"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89\": rpc error: code = NotFound desc = could not find container \"875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89\": container with ID starting with 875501749d15bd21beac318ac1766453e438f2fcde3e762cb12297f302ee2f89 not found: ID does not exist" May 20 14:03:53.426225 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.426211 2575 scope.go:117] "RemoveContainer" containerID="d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a" May 20 14:03:53.426386 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:53.426371 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a\": container with ID starting with d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a not found: ID does not exist" containerID="d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a" May 20 14:03:53.426424 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.426390 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a"} err="failed to get container status \"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a\": rpc error: code = NotFound desc = could not find container \"d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a\": container with ID starting with d37e7ad1bdccbe6de454aad39e593dc4fcad440c3390b69b57d1a43ad0c0105a not found: ID does not exist" May 20 14:03:53.482415 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482382 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert\") pod \"3dd098d1-0850-4e94-8fba-0197f4b1f733\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " May 20 14:03:53.482415 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482428 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location\") pod \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482456 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"3dd098d1-0850-4e94-8fba-0197f4b1f733\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482485 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location\") pod \"3dd098d1-0850-4e94-8fba-0197f4b1f733\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482503 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls\") pod \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482521 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert\") pod \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482548 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv98h\" (UniqueName: \"kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h\") pod \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " May 20 14:03:53.482652 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482576 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\" (UID: \"d35c4ca2-5c86-491f-8694-a7904ffd2bc8\") " May 20 14:03:53.482973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482841 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dd098d1-0850-4e94-8fba-0197f4b1f733" (UID: "3dd098d1-0850-4e94-8fba-0197f4b1f733"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:03:53.482973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482858 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d35c4ca2-5c86-491f-8694-a7904ffd2bc8" (UID: "d35c4ca2-5c86-491f-8694-a7904ffd2bc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:03:53.482973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482875 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dd098d1-0850-4e94-8fba-0197f4b1f733" (UID: "3dd098d1-0850-4e94-8fba-0197f4b1f733"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:53.482973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482925 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d35c4ca2-5c86-491f-8694-a7904ffd2bc8" (UID: "d35c4ca2-5c86-491f-8694-a7904ffd2bc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:53.482973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.482954 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "3dd098d1-0850-4e94-8fba-0197f4b1f733" (UID: "3dd098d1-0850-4e94-8fba-0197f4b1f733"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:53.483214 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483001 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f76m\" (UniqueName: \"kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m\") pod \"3dd098d1-0850-4e94-8fba-0197f4b1f733\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " May 20 14:03:53.483214 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483054 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls\") pod \"3dd098d1-0850-4e94-8fba-0197f4b1f733\" (UID: \"3dd098d1-0850-4e94-8fba-0197f4b1f733\") " May 20 14:03:53.483214 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483199 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config") pod "d35c4ca2-5c86-491f-8694-a7904ffd2bc8" (UID: "d35c4ca2-5c86-491f-8694-a7904ffd2bc8"). InnerVolumeSpecName "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:53.483362 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483341 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-cabundle-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.483400 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483370 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kserve-provision-location\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.483400 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483388 2575 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dd098d1-0850-4e94-8fba-0197f4b1f733-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.483469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483403 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dd098d1-0850-4e94-8fba-0197f4b1f733-kserve-provision-location\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.483469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483418 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-cabundle-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.483469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.483433 2575 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.484977 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.484955 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dd098d1-0850-4e94-8fba-0197f4b1f733" (UID: "3dd098d1-0850-4e94-8fba-0197f4b1f733"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:53.485053 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.484995 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m" (OuterVolumeSpecName: "kube-api-access-7f76m") pod "3dd098d1-0850-4e94-8fba-0197f4b1f733" (UID: "3dd098d1-0850-4e94-8fba-0197f4b1f733"). InnerVolumeSpecName "kube-api-access-7f76m". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:53.485173 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.485156 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d35c4ca2-5c86-491f-8694-a7904ffd2bc8" (UID: "d35c4ca2-5c86-491f-8694-a7904ffd2bc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:53.485210 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.485149 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h" (OuterVolumeSpecName: "kube-api-access-dv98h") pod "d35c4ca2-5c86-491f-8694-a7904ffd2bc8" (UID: "d35c4ca2-5c86-491f-8694-a7904ffd2bc8"). InnerVolumeSpecName "kube-api-access-dv98h". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:53.584794 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.584728 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.584794 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.584789 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv98h\" (UniqueName: \"kubernetes.io/projected/d35c4ca2-5c86-491f-8694-a7904ffd2bc8-kube-api-access-dv98h\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.584794 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.584801 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f76m\" (UniqueName: \"kubernetes.io/projected/3dd098d1-0850-4e94-8fba-0197f4b1f733-kube-api-access-7f76m\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.585017 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.584810 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dd098d1-0850-4e94-8fba-0197f4b1f733-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:53.725994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.725956 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:03:53.730643 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.730616 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-58cc67cb8-c6bdg"] May 20 14:03:53.846976 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:53.846936 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" path="/var/lib/kubelet/pods/d35c4ca2-5c86-491f-8694-a7904ffd2bc8/volumes" May 20 14:03:54.407815 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.407775 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" event={"ID":"3dd098d1-0850-4e94-8fba-0197f4b1f733","Type":"ContainerDied","Data":"ab98a3fd87e43b8c53292c328b2a923dfc4516f250ea098e582e1ada44ae7124"} May 20 14:03:54.408239 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.407831 2575 scope.go:117] "RemoveContainer" containerID="bc4ecd50c8569165b6eeb85d3a108b18f912a0de68e8774c7396e1f796328d07" May 20 14:03:54.408239 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.407859 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r" May 20 14:03:54.415740 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.415721 2575 scope.go:117] "RemoveContainer" containerID="4f47f767ba0b4f1ccb6dc9bbecbccf25787f5af0339996747acf97bb2365e812" May 20 14:03:54.422810 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.422791 2575 scope.go:117] "RemoveContainer" containerID="8ddb55715fc3e7f4607d76f56d265c29779c84df49685268f02b0261b19d920d" May 20 14:03:54.445583 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.441971 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:03:54.448868 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:54.448834 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-xhv8r"] May 20 14:03:55.842106 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:55.842064 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" path="/var/lib/kubelet/pods/3dd098d1-0850-4e94-8fba-0197f4b1f733/volumes" May 20 14:03:56.039673 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.039633 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused" May 20 14:03:56.044997 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.044965 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused" May 20 14:03:56.811048 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.811026 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:03:56.910894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.910806 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n86hh\" (UniqueName: \"kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh\") pod \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " May 20 14:03:56.910894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.910854 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location\") pod \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " May 20 14:03:56.910894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.910875 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " May 20 14:03:56.911440 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.910913 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert\") pod \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " May 20 14:03:56.911440 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.910965 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") pod \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\" (UID: \"efd1378d-8fc2-43b0-9a18-4cce7fac91f9\") " May 20 14:03:56.911440 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.911253 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efd1378d-8fc2-43b0-9a18-4cce7fac91f9" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:03:56.911440 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.911313 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "efd1378d-8fc2-43b0-9a18-4cce7fac91f9" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:56.911440 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.911352 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efd1378d-8fc2-43b0-9a18-4cce7fac91f9" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:56.913057 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.913037 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efd1378d-8fc2-43b0-9a18-4cce7fac91f9" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:56.913111 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:56.913085 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh" (OuterVolumeSpecName: "kube-api-access-n86hh") pod "efd1378d-8fc2-43b0-9a18-4cce7fac91f9" (UID: "efd1378d-8fc2-43b0-9a18-4cce7fac91f9"). InnerVolumeSpecName "kube-api-access-n86hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:57.012252 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.012214 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:57.012252 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.012247 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86hh\" (UniqueName: \"kubernetes.io/projected/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kube-api-access-n86hh\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:57.012252 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.012257 2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-kserve-provision-location\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:57.012477 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.012266 2575 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:57.012477 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.012278 2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efd1378d-8fc2-43b0-9a18-4cce7fac91f9-cabundle-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:03:57.307467 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.307381 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused" May 20 14:03:57.400339 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.400310 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:03:57.400892 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.400869 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:03:57.419171 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.419143 2575 generic.go:358] "Generic (PLEG): container finished" podID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerID="86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe" exitCode=0 May 20 14:03:57.419311 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.419226 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" May 20 14:03:57.419311 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.419223 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerDied","Data":"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe"} May 20 14:03:57.419311 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.419271 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d" event={"ID":"efd1378d-8fc2-43b0-9a18-4cce7fac91f9","Type":"ContainerDied","Data":"ff83eb3465d908a279c17f778b570961f68d288b172171688cc7a80fd790b959"} May 20 14:03:57.419311 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.419291 2575 scope.go:117] "RemoveContainer" containerID="e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2" May 20 14:03:57.428303 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.428276 2575 scope.go:117] "RemoveContainer" containerID="86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe" May 20 14:03:57.435538 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.435520 2575 scope.go:117] "RemoveContainer" containerID="5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd" May 20 14:03:57.442156 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442134 2575 scope.go:117] "RemoveContainer" containerID="e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2" May 20 14:03:57.442362 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:57.442346 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2\": container with ID starting with e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2 not found: ID does not exist" containerID="e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2" May 20 14:03:57.442472 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442370 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2"} err="failed to get container status \"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2\": rpc error: code = NotFound desc = could not find container \"e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2\": container with ID starting with e36254a352dbdf957c81d623126e944d18870dbab7e1ade67d7882360b1569b2 not found: ID does not exist" May 20 14:03:57.442472 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442386 2575 scope.go:117] "RemoveContainer" containerID="86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe" May 20 14:03:57.442597 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:57.442576 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe\": container with ID starting with 86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe not found: ID does not exist" containerID="86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe" May 20 14:03:57.442637 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442603 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe"} err="failed to get container status \"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe\": rpc error: code = NotFound desc = could not find container \"86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe\": container with ID starting with 86810a685e63d3449db5a4be56e09ba646b80d1de7cc1ff5e64d0a5419e002fe not found: ID does not exist" May 20 14:03:57.442637 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442619 2575 scope.go:117] "RemoveContainer" containerID="5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd" May 20 14:03:57.442877 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:03:57.442859 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd\": container with ID starting with 5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd not found: ID does not exist" containerID="5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd" May 20 14:03:57.442939 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.442884 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd"} err="failed to get container status \"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd\": rpc error: code = NotFound desc = could not find container \"5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd\": container with ID starting with 5c62c9dd2cfd17f8daa2e850d343eb31e367bfee58fa8b882cc6a4b4ab667cdd not found: ID does not exist" May 20 14:03:57.445962 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.445939 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:03:57.450994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.450972 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-664c5c98bd-8nr4d"] May 20 14:03:57.843514 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:03:57.843485 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" path="/var/lib/kubelet/pods/efd1378d-8fc2-43b0-9a18-4cce7fac91f9/volumes" May 20 14:04:07.307414 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:04:07.307383 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:04:07.401695 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:04:07.401659 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:04:17.400934 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:04:17.400887 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:04:27.401008 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:04:27.400964 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" May 20 14:04:37.402396 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:04:37.402359 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:12:33.664546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.664512 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:12:33.667315 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.664798 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" containerID="cri-o://68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3" gracePeriod=30 May 20 14:12:33.667315 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.664839 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kube-rbac-proxy" containerID="cri-o://8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371" gracePeriod=30 May 20 14:12:33.853512 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853480 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:12:33.853783 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853771 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853785 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853795 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kube-rbac-proxy" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853800 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kube-rbac-proxy" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853811 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853816 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853825 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853830 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" May 20 14:12:33.853838 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853838 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853844 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853854 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kube-rbac-proxy" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853859 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kube-rbac-proxy" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853866 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853871 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853876 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853882 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853890 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853895 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="storage-initializer" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853935 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kube-rbac-proxy" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853943 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kube-rbac-proxy" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853950 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kube-rbac-proxy" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853959 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="efd1378d-8fc2-43b0-9a18-4cce7fac91f9" containerName="kserve-container" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853964 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="3dd098d1-0850-4e94-8fba-0197f4b1f733" containerName="kserve-container" May 20 14:12:33.854091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.853971 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="d35c4ca2-5c86-491f-8694-a7904ffd2bc8" containerName="kserve-container" May 20 14:12:33.856842 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.856825 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:33.872421 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.872398 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-15cdb-predictor-serving-cert\"" May 20 14:12:33.872543 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.872431 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\"" May 20 14:12:33.886823 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.886794 2575 generic.go:358] "Generic (PLEG): container finished" podID="5b59113e-1550-4804-b847-d4e3d67f7931" containerID="8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371" exitCode=2 May 20 14:12:33.886918 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.886844 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerDied","Data":"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371"} May 20 14:12:33.902629 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.902597 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:12:33.962509 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.962424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:33.962644 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.962583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznpd\" (UniqueName: \"kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:33.962721 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:33.962705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.063382 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.063350 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.063526 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.063432 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.063526 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.063451 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qznpd\" (UniqueName: \"kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.064155 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.064131 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.065873 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.065851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.126325 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.126301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznpd\" (UniqueName: \"kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd\") pod \"success-200-isvc-15cdb-predictor-679cfbb586-s4rql\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.166780 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.166736 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.301703 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:12:34.301671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc9b081_db86_4142_ac1e_c60c7de5d7cf.slice/crio-ea66b0d7feacc64e1e9bdeed9018f6f936629ac63be62d508d3dc39f6072700e WatchSource:0}: Error finding container ea66b0d7feacc64e1e9bdeed9018f6f936629ac63be62d508d3dc39f6072700e: Status 404 returned error can't find the container with id ea66b0d7feacc64e1e9bdeed9018f6f936629ac63be62d508d3dc39f6072700e May 20 14:12:34.303214 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.303189 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:12:34.303876 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.303862 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:12:34.890596 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.890561 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerStarted","Data":"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b"} May 20 14:12:34.890596 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.890597 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerStarted","Data":"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4"} May 20 14:12:34.891060 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.890607 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerStarted","Data":"ea66b0d7feacc64e1e9bdeed9018f6f936629ac63be62d508d3dc39f6072700e"} May 20 14:12:34.891060 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.890769 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.891060 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.890790 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:34.892133 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.892108 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:12:34.922152 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:34.922110 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podStartSLOduration=1.92209586 podStartE2EDuration="1.92209586s" podCreationTimestamp="2026-05-20 14:12:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:12:34.921165422 +0000 UTC m=+1339.671977820" watchObservedRunningTime="2026-05-20 14:12:34.92209586 +0000 UTC m=+1339.672908258" May 20 14:12:35.893452 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:35.893416 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:12:36.801393 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.801373 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:12:36.888159 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.888132 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdgxx\" (UniqueName: \"kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx\") pod \"5b59113e-1550-4804-b847-d4e3d67f7931\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " May 20 14:12:36.888299 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.888176 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config\") pod \"5b59113e-1550-4804-b847-d4e3d67f7931\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " May 20 14:12:36.888299 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.888208 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") pod \"5b59113e-1550-4804-b847-d4e3d67f7931\" (UID: \"5b59113e-1550-4804-b847-d4e3d67f7931\") " May 20 14:12:36.888559 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.888529 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-83116-kube-rbac-proxy-sar-config") pod "5b59113e-1550-4804-b847-d4e3d67f7931" (UID: "5b59113e-1550-4804-b847-d4e3d67f7931"). InnerVolumeSpecName "success-200-isvc-83116-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:12:36.890231 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.890206 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b59113e-1550-4804-b847-d4e3d67f7931" (UID: "5b59113e-1550-4804-b847-d4e3d67f7931"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:12:36.890336 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.890239 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx" (OuterVolumeSpecName: "kube-api-access-fdgxx") pod "5b59113e-1550-4804-b847-d4e3d67f7931" (UID: "5b59113e-1550-4804-b847-d4e3d67f7931"). InnerVolumeSpecName "kube-api-access-fdgxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:12:36.897730 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.897705 2575 generic.go:358] "Generic (PLEG): container finished" podID="5b59113e-1550-4804-b847-d4e3d67f7931" containerID="68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3" exitCode=0 May 20 14:12:36.898113 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.897766 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerDied","Data":"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3"} May 20 14:12:36.898113 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.897801 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" event={"ID":"5b59113e-1550-4804-b847-d4e3d67f7931","Type":"ContainerDied","Data":"2437c32337df9a12da48fa7ac005f3ff85839a3e0c627e401f960076688ffa67"} May 20 14:12:36.898113 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.897823 2575 scope.go:117] "RemoveContainer" containerID="8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371" May 20 14:12:36.898113 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.897827 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms" May 20 14:12:36.906532 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.906515 2575 scope.go:117] "RemoveContainer" containerID="68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3" May 20 14:12:36.913372 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.913359 2575 scope.go:117] "RemoveContainer" containerID="8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371" May 20 14:12:36.913647 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:12:36.913630 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371\": container with ID starting with 8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371 not found: ID does not exist" containerID="8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371" May 20 14:12:36.913725 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.913659 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371"} err="failed to get container status \"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371\": rpc error: code = NotFound desc = could not find container \"8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371\": container with ID starting with 8e00afc9b473fef86ed94c871a54b8a8be691d17a86d312ace547743a852b371 not found: ID does not exist" May 20 14:12:36.913725 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.913684 2575 scope.go:117] "RemoveContainer" containerID="68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3" May 20 14:12:36.913976 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:12:36.913958 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3\": container with ID starting with 68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3 not found: ID does not exist" containerID="68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3" May 20 14:12:36.914028 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.913980 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3"} err="failed to get container status \"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3\": rpc error: code = NotFound desc = could not find container \"68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3\": container with ID starting with 68fa3e3d408bd878d2f32de4cc4e1dc32169c072a992e989cf9186c55ae360b3 not found: ID does not exist" May 20 14:12:36.923156 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.923136 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:12:36.929022 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.929000 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83116-predictor-59d5fb4b86-m27ms"] May 20 14:12:36.989243 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.989220 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-83116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b59113e-1550-4804-b847-d4e3d67f7931-success-200-isvc-83116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:12:36.989243 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.989241 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b59113e-1550-4804-b847-d4e3d67f7931-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:12:36.989378 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:36.989252 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdgxx\" (UniqueName: \"kubernetes.io/projected/5b59113e-1550-4804-b847-d4e3d67f7931-kube-api-access-fdgxx\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:12:37.842328 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:37.842286 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" path="/var/lib/kubelet/pods/5b59113e-1550-4804-b847-d4e3d67f7931/volumes" May 20 14:12:40.897713 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:40.897682 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:12:40.898233 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:40.898212 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:12:50.898640 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:12:50.898604 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:13:00.898982 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:00.898938 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:13:03.422638 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.422606 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:13:03.423014 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.422884 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" containerID="cri-o://b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3" gracePeriod=30 May 20 14:13:03.423014 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.422922 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kube-rbac-proxy" containerID="cri-o://51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85" gracePeriod=30 May 20 14:13:03.452846 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.452814 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:13:03.453149 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453135 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" May 20 14:13:03.453149 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453149 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" May 20 14:13:03.453275 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453168 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kube-rbac-proxy" May 20 14:13:03.453275 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453174 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kube-rbac-proxy" May 20 14:13:03.453275 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453220 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kube-rbac-proxy" May 20 14:13:03.453275 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.453230 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b59113e-1550-4804-b847-d4e3d67f7931" containerName="kserve-container" May 20 14:13:03.456000 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.455984 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.458637 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.458618 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-14a90-kube-rbac-proxy-sar-config\"" May 20 14:13:03.458724 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.458668 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-14a90-predictor-serving-cert\"" May 20 14:13:03.486534 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.486508 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:13:03.494466 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.494429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.494604 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.494556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.494673 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.494634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47l9s\" (UniqueName: \"kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.596026 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.595990 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.596198 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.596074 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47l9s\" (UniqueName: \"kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.596198 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.596115 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.596731 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.596705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.598541 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.598518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.622853 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.622780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47l9s\" (UniqueName: \"kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s\") pod \"success-200-isvc-14a90-predictor-75bc8f9bb-fv57c\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.766162 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.766065 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:03.904651 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.904547 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:13:03.907187 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:13:03.907161 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78117ed5_88b7_4c96_8676_0f39b68f132d.slice/crio-8c10a563a8d2b4e969922212519b4a38a7a202bf795bde517afe25f72b35116a WatchSource:0}: Error finding container 8c10a563a8d2b4e969922212519b4a38a7a202bf795bde517afe25f72b35116a: Status 404 returned error can't find the container with id 8c10a563a8d2b4e969922212519b4a38a7a202bf795bde517afe25f72b35116a May 20 14:13:03.979196 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.979172 2575 generic.go:358] "Generic (PLEG): container finished" podID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerID="51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85" exitCode=2 May 20 14:13:03.979288 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.979249 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerDied","Data":"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85"} May 20 14:13:03.980498 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.980477 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerStarted","Data":"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e"} May 20 14:13:03.980579 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:03.980506 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerStarted","Data":"8c10a563a8d2b4e969922212519b4a38a7a202bf795bde517afe25f72b35116a"} May 20 14:13:04.984685 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:04.984653 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerStarted","Data":"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3"} May 20 14:13:04.985080 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:04.984774 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:05.004531 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:05.004480 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podStartSLOduration=2.004463959 podStartE2EDuration="2.004463959s" podCreationTimestamp="2026-05-20 14:13:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:05.004201914 +0000 UTC m=+1369.755014316" watchObservedRunningTime="2026-05-20 14:13:05.004463959 +0000 UTC m=+1369.755276359" May 20 14:13:05.987735 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:05.987705 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:05.989075 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:05.989048 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:06.567872 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.567850 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:13:06.622418 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.622395 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\") pod \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " May 20 14:13:06.622557 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.622453 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4swxh\" (UniqueName: \"kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh\") pod \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " May 20 14:13:06.622557 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.622520 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") pod \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\" (UID: \"c89fd641-6384-45fc-9a9b-eb65d310c2e3\") " May 20 14:13:06.622799 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.622771 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-cdd1d-kube-rbac-proxy-sar-config") pod "c89fd641-6384-45fc-9a9b-eb65d310c2e3" (UID: "c89fd641-6384-45fc-9a9b-eb65d310c2e3"). InnerVolumeSpecName "success-200-isvc-cdd1d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:13:06.624504 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.624474 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c89fd641-6384-45fc-9a9b-eb65d310c2e3" (UID: "c89fd641-6384-45fc-9a9b-eb65d310c2e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:13:06.624575 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.624556 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh" (OuterVolumeSpecName: "kube-api-access-4swxh") pod "c89fd641-6384-45fc-9a9b-eb65d310c2e3" (UID: "c89fd641-6384-45fc-9a9b-eb65d310c2e3"). InnerVolumeSpecName "kube-api-access-4swxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:13:06.723236 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.723208 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c89fd641-6384-45fc-9a9b-eb65d310c2e3-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:06.723236 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.723232 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c89fd641-6384-45fc-9a9b-eb65d310c2e3-success-200-isvc-cdd1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:06.723236 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.723243 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swxh\" (UniqueName: \"kubernetes.io/projected/c89fd641-6384-45fc-9a9b-eb65d310c2e3-kube-api-access-4swxh\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:06.991712 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.991627 2575 generic.go:358] "Generic (PLEG): container finished" podID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerID="b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3" exitCode=0 May 20 14:13:06.991712 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.991697 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" May 20 14:13:06.992220 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.991711 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerDied","Data":"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3"} May 20 14:13:06.992220 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.991769 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv" event={"ID":"c89fd641-6384-45fc-9a9b-eb65d310c2e3","Type":"ContainerDied","Data":"087946862fb563192040f866a72a75888eca6c60365072e40f149598080cb1e9"} May 20 14:13:06.992220 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.991793 2575 scope.go:117] "RemoveContainer" containerID="51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85" May 20 14:13:06.992337 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.992219 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:07.000022 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:06.999975 2575 scope.go:117] "RemoveContainer" containerID="b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3" May 20 14:13:07.006671 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.006655 2575 scope.go:117] "RemoveContainer" containerID="51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85" May 20 14:13:07.006917 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:13:07.006899 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85\": container with ID starting with 51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85 not found: ID does not exist" containerID="51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85" May 20 14:13:07.006978 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.006924 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85"} err="failed to get container status \"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85\": rpc error: code = NotFound desc = could not find container \"51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85\": container with ID starting with 51b933e54477be16e8ec30c390b17750023121dba6dddc11bb5f66d1949e5e85 not found: ID does not exist" May 20 14:13:07.006978 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.006942 2575 scope.go:117] "RemoveContainer" containerID="b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3" May 20 14:13:07.007176 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:13:07.007157 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3\": container with ID starting with b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3 not found: ID does not exist" containerID="b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3" May 20 14:13:07.007225 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.007183 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3"} err="failed to get container status \"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3\": rpc error: code = NotFound desc = could not find container \"b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3\": container with ID starting with b3619adba97d5fd8223dc7b87ca7de8a7f49584a129d38b5f91cdcc170d6ffb3 not found: ID does not exist" May 20 14:13:07.033788 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.033760 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:13:07.046672 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.046643 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cdd1d-predictor-685c7f8cd8-5jgsv"] May 20 14:13:07.842163 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:07.842133 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" path="/var/lib/kubelet/pods/c89fd641-6384-45fc-9a9b-eb65d310c2e3/volumes" May 20 14:13:10.898761 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:10.898704 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" May 20 14:13:11.996675 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:11.996649 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:13:11.997219 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:11.997192 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:20.898977 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:20.898948 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:13:21.997972 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:21.997933 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:31.998247 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:31.998208 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:41.998114 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:41.998077 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" May 20 14:13:43.901604 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.901562 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:13:43.901973 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.901947 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" containerID="cri-o://d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4" gracePeriod=30 May 20 14:13:43.902037 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.901989 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kube-rbac-proxy" containerID="cri-o://cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b" gracePeriod=30 May 20 14:13:43.932488 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932456 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:13:43.932847 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932830 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" May 20 14:13:43.932929 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932849 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" May 20 14:13:43.932929 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932861 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kube-rbac-proxy" May 20 14:13:43.932929 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932869 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kube-rbac-proxy" May 20 14:13:43.933086 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932963 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kube-rbac-proxy" May 20 14:13:43.933086 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.932980 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="c89fd641-6384-45fc-9a9b-eb65d310c2e3" containerName="kserve-container" May 20 14:13:43.936007 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.935989 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:43.938461 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.938440 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ecafe-predictor-serving-cert\"" May 20 14:13:43.940254 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:43.940240 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\"" May 20 14:13:44.009057 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.009029 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:13:44.037490 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.037461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.037606 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.037501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.037650 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.037616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhx2q\" (UniqueName: \"kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.094376 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.094348 2575 generic.go:358] "Generic (PLEG): container finished" podID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerID="cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b" exitCode=2 May 20 14:13:44.094533 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.094411 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerDied","Data":"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b"} May 20 14:13:44.138175 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.138148 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhx2q\" (UniqueName: \"kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.138313 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.138207 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.138313 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.138227 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.138886 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.138867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.140499 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.140480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.149863 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.149842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhx2q\" (UniqueName: \"kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q\") pod \"success-200-isvc-ecafe-predictor-785fb74b88-nx9dq\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.246175 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.246091 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:44.371407 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:44.371373 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:13:44.373725 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:13:44.373697 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1303af_b906_402f_9ba8_44501594a4b4.slice/crio-4adcc2dafb67705997e4fd6421cf8ffd7584e933703e98100d106110a03bb7d1 WatchSource:0}: Error finding container 4adcc2dafb67705997e4fd6421cf8ffd7584e933703e98100d106110a03bb7d1: Status 404 returned error can't find the container with id 4adcc2dafb67705997e4fd6421cf8ffd7584e933703e98100d106110a03bb7d1 May 20 14:13:45.098815 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.098779 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerStarted","Data":"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6"} May 20 14:13:45.098815 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.098818 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerStarted","Data":"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e"} May 20 14:13:45.099235 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.098833 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerStarted","Data":"4adcc2dafb67705997e4fd6421cf8ffd7584e933703e98100d106110a03bb7d1"} May 20 14:13:45.099235 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.098863 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:45.099235 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.098961 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:45.100205 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.100180 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:13:45.118689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.118643 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podStartSLOduration=2.118631637 podStartE2EDuration="2.118631637s" podCreationTimestamp="2026-05-20 14:13:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:45.11720548 +0000 UTC m=+1409.868017878" watchObservedRunningTime="2026-05-20 14:13:45.118631637 +0000 UTC m=+1409.869444034" May 20 14:13:45.894283 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:45.894245 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused" May 20 14:13:46.101994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:46.101959 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:13:47.051060 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.051036 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:13:47.060898 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.060879 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznpd\" (UniqueName: \"kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd\") pod \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " May 20 14:13:47.060994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.060925 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls\") pod \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " May 20 14:13:47.060994 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.060953 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config\") pod \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\" (UID: \"bcc9b081-db86-4142-ac1e-c60c7de5d7cf\") " May 20 14:13:47.061359 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.061333 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-15cdb-kube-rbac-proxy-sar-config") pod "bcc9b081-db86-4142-ac1e-c60c7de5d7cf" (UID: "bcc9b081-db86-4142-ac1e-c60c7de5d7cf"). InnerVolumeSpecName "success-200-isvc-15cdb-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:13:47.062849 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.062826 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd" (OuterVolumeSpecName: "kube-api-access-qznpd") pod "bcc9b081-db86-4142-ac1e-c60c7de5d7cf" (UID: "bcc9b081-db86-4142-ac1e-c60c7de5d7cf"). InnerVolumeSpecName "kube-api-access-qznpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:13:47.062943 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.062866 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcc9b081-db86-4142-ac1e-c60c7de5d7cf" (UID: "bcc9b081-db86-4142-ac1e-c60c7de5d7cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:13:47.105406 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.105335 2575 generic.go:358] "Generic (PLEG): container finished" podID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerID="d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4" exitCode=0 May 20 14:13:47.105726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.105404 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" May 20 14:13:47.105726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.105421 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerDied","Data":"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4"} May 20 14:13:47.105726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.105459 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql" event={"ID":"bcc9b081-db86-4142-ac1e-c60c7de5d7cf","Type":"ContainerDied","Data":"ea66b0d7feacc64e1e9bdeed9018f6f936629ac63be62d508d3dc39f6072700e"} May 20 14:13:47.105726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.105478 2575 scope.go:117] "RemoveContainer" containerID="cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b" May 20 14:13:47.113569 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.113551 2575 scope.go:117] "RemoveContainer" containerID="d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4" May 20 14:13:47.120394 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.120378 2575 scope.go:117] "RemoveContainer" containerID="cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b" May 20 14:13:47.120597 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:13:47.120579 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b\": container with ID starting with cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b not found: ID does not exist" containerID="cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b" May 20 14:13:47.120654 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.120609 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b"} err="failed to get container status \"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b\": rpc error: code = NotFound desc = could not find container \"cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b\": container with ID starting with cb8b74a31237b8272b8cc16d070b1280defd7e60887a76af66ebacccd41c2c0b not found: ID does not exist" May 20 14:13:47.120654 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.120627 2575 scope.go:117] "RemoveContainer" containerID="d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4" May 20 14:13:47.120882 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:13:47.120866 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4\": container with ID starting with d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4 not found: ID does not exist" containerID="d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4" May 20 14:13:47.120942 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.120885 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4"} err="failed to get container status \"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4\": rpc error: code = NotFound desc = could not find container \"d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4\": container with ID starting with d80c7d3b36919471f8041a35688654114e73f5c9037811713eef9658edc205c4 not found: ID does not exist" May 20 14:13:47.129010 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.128990 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:13:47.135869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.135849 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-15cdb-predictor-679cfbb586-s4rql"] May 20 14:13:47.161996 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.161976 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznpd\" (UniqueName: \"kubernetes.io/projected/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-kube-api-access-qznpd\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:47.162079 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.162000 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:47.162079 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.162016 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-15cdb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcc9b081-db86-4142-ac1e-c60c7de5d7cf-success-200-isvc-15cdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:13:47.842677 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:47.842645 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" path="/var/lib/kubelet/pods/bcc9b081-db86-4142-ac1e-c60c7de5d7cf/volumes" May 20 14:13:51.105945 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:51.105919 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:13:51.106378 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:51.106354 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:13:51.998513 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:13:51.998486 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:14:01.107127 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:01.107089 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:14:11.107098 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:11.107010 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:14:13.687673 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.687643 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.687975 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kube-rbac-proxy" May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.687987 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kube-rbac-proxy" May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.688002 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.688007 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.688054 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kserve-container" May 20 14:14:13.688077 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.688063 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcc9b081-db86-4142-ac1e-c60c7de5d7cf" containerName="kube-rbac-proxy" May 20 14:14:13.690927 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.690911 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.696295 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.696274 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-734fc-predictor-serving-cert\"" May 20 14:14:13.696767 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.696735 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-734fc-kube-rbac-proxy-sar-config\"" May 20 14:14:13.703049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.703029 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:14:13.710133 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.710110 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:14:13.710364 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.710345 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" containerID="cri-o://6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e" gracePeriod=30 May 20 14:14:13.710450 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.710424 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kube-rbac-proxy" containerID="cri-o://41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3" gracePeriod=30 May 20 14:14:13.768498 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.768471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.768600 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.768517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6ln\" (UniqueName: \"kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.768642 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.768596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.869018 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.868987 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.869152 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.869045 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.869152 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.869077 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6ln\" (UniqueName: \"kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.869152 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:14:13.869126 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-734fc-predictor-serving-cert: secret "success-200-isvc-734fc-predictor-serving-cert" not found May 20 14:14:13.869324 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:14:13.869198 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls podName:d70962d5-e2e1-45e6-b610-376b13dc6648 nodeName:}" failed. No retries permitted until 2026-05-20 14:14:14.369176956 +0000 UTC m=+1439.119989333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls") pod "success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" (UID: "d70962d5-e2e1-45e6-b610-376b13dc6648") : secret "success-200-isvc-734fc-predictor-serving-cert" not found May 20 14:14:13.869674 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.869650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:13.886977 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:13.886951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6ln\" (UniqueName: \"kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:14.184049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.184020 2575 generic.go:358] "Generic (PLEG): container finished" podID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerID="41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3" exitCode=2 May 20 14:14:14.184198 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.184084 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerDied","Data":"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3"} May 20 14:14:14.373671 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.373642 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:14.375961 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.375940 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") pod \"success-200-isvc-734fc-predictor-64d98dd8b6-bknhg\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:14.601514 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.601428 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:14.723861 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:14.723800 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:14:14.726389 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:14:14.726360 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70962d5_e2e1_45e6_b610_376b13dc6648.slice/crio-d6cdd83bec9edca48b98e2e6ad44f6b9125b7506d4e0b1b3b27e83f3b58c4ecc WatchSource:0}: Error finding container d6cdd83bec9edca48b98e2e6ad44f6b9125b7506d4e0b1b3b27e83f3b58c4ecc: Status 404 returned error can't find the container with id d6cdd83bec9edca48b98e2e6ad44f6b9125b7506d4e0b1b3b27e83f3b58c4ecc May 20 14:14:15.188974 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:15.188936 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerStarted","Data":"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48"} May 20 14:14:15.188974 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:15.188977 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerStarted","Data":"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb"} May 20 14:14:15.189240 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:15.188987 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerStarted","Data":"d6cdd83bec9edca48b98e2e6ad44f6b9125b7506d4e0b1b3b27e83f3b58c4ecc"} May 20 14:14:15.189240 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:15.189057 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:15.211567 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:15.211515 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podStartSLOduration=2.211500366 podStartE2EDuration="2.211500366s" podCreationTimestamp="2026-05-20 14:14:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:14:15.210149858 +0000 UTC m=+1439.960962256" watchObservedRunningTime="2026-05-20 14:14:15.211500366 +0000 UTC m=+1439.962312764" May 20 14:14:16.191969 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.191933 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:16.193242 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.193215 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:14:16.949755 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.949719 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:14:16.997610 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.997535 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config\") pod \"78117ed5-88b7-4c96-8676-0f39b68f132d\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " May 20 14:14:16.997610 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.997571 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47l9s\" (UniqueName: \"kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s\") pod \"78117ed5-88b7-4c96-8676-0f39b68f132d\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " May 20 14:14:16.997610 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.997610 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls\") pod \"78117ed5-88b7-4c96-8676-0f39b68f132d\" (UID: \"78117ed5-88b7-4c96-8676-0f39b68f132d\") " May 20 14:14:16.997915 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.997892 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-14a90-kube-rbac-proxy-sar-config") pod "78117ed5-88b7-4c96-8676-0f39b68f132d" (UID: "78117ed5-88b7-4c96-8676-0f39b68f132d"). InnerVolumeSpecName "success-200-isvc-14a90-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:14:16.999650 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.999629 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78117ed5-88b7-4c96-8676-0f39b68f132d" (UID: "78117ed5-88b7-4c96-8676-0f39b68f132d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:14:16.999650 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:16.999629 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s" (OuterVolumeSpecName: "kube-api-access-47l9s") pod "78117ed5-88b7-4c96-8676-0f39b68f132d" (UID: "78117ed5-88b7-4c96-8676-0f39b68f132d"). InnerVolumeSpecName "kube-api-access-47l9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:14:17.098147 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.098112 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-14a90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78117ed5-88b7-4c96-8676-0f39b68f132d-success-200-isvc-14a90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:14:17.098147 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.098142 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47l9s\" (UniqueName: \"kubernetes.io/projected/78117ed5-88b7-4c96-8676-0f39b68f132d-kube-api-access-47l9s\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:14:17.098147 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.098153 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78117ed5-88b7-4c96-8676-0f39b68f132d-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:14:17.195657 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.195621 2575 generic.go:358] "Generic (PLEG): container finished" podID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerID="6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e" exitCode=0 May 20 14:14:17.196092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.195701 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerDied","Data":"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e"} May 20 14:14:17.196092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.195710 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" May 20 14:14:17.196092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.195760 2575 scope.go:117] "RemoveContainer" containerID="41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3" May 20 14:14:17.196092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.195735 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c" event={"ID":"78117ed5-88b7-4c96-8676-0f39b68f132d","Type":"ContainerDied","Data":"8c10a563a8d2b4e969922212519b4a38a7a202bf795bde517afe25f72b35116a"} May 20 14:14:17.196092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.196078 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:14:17.204383 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.204188 2575 scope.go:117] "RemoveContainer" containerID="6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e" May 20 14:14:17.212842 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.212824 2575 scope.go:117] "RemoveContainer" containerID="41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3" May 20 14:14:17.213072 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:14:17.213056 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3\": container with ID starting with 41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3 not found: ID does not exist" containerID="41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3" May 20 14:14:17.213120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.213080 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3"} err="failed to get container status \"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3\": rpc error: code = NotFound desc = could not find container \"41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3\": container with ID starting with 41e261f3505d87a2cc2f6a86d45788e36f71af76f5d9a5a44dc5c6a9d5f018b3 not found: ID does not exist" May 20 14:14:17.213120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.213095 2575 scope.go:117] "RemoveContainer" containerID="6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e" May 20 14:14:17.213304 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:14:17.213290 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e\": container with ID starting with 6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e not found: ID does not exist" containerID="6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e" May 20 14:14:17.213340 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.213308 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e"} err="failed to get container status \"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e\": rpc error: code = NotFound desc = could not find container \"6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e\": container with ID starting with 6cf60227f64c523942c79ea413f439c681fea08e83219f563e73e072ac02e99e not found: ID does not exist" May 20 14:14:17.225433 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.225410 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:14:17.234281 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.234262 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14a90-predictor-75bc8f9bb-fv57c"] May 20 14:14:17.842689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:17.842659 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" path="/var/lib/kubelet/pods/78117ed5-88b7-4c96-8676-0f39b68f132d/volumes" May 20 14:14:21.106433 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:21.106399 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:14:22.200923 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:22.200890 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:14:22.201485 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:22.201461 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:14:31.106913 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:31.106884 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:14:32.202156 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:32.202118 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:14:42.202055 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:42.202016 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:14:52.202034 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:14:52.201992 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" May 20 14:15:02.201929 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:15:02.201894 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:22:58.753045 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.753016 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:22:58.755569 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.753389 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" containerID="cri-o://2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e" gracePeriod=30 May 20 14:22:58.755569 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.753440 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kube-rbac-proxy" containerID="cri-o://13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6" gracePeriod=30 May 20 14:22:58.805138 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805081 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:22:58.805536 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805521 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kube-rbac-proxy" May 20 14:22:58.805605 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805539 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kube-rbac-proxy" May 20 14:22:58.805605 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805562 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" May 20 14:22:58.805605 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805570 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" May 20 14:22:58.805725 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805664 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kube-rbac-proxy" May 20 14:22:58.805725 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.805678 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="78117ed5-88b7-4c96-8676-0f39b68f132d" containerName="kserve-container" May 20 14:22:58.808875 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.808858 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.811423 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.811273 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a868a-kube-rbac-proxy-sar-config\"" May 20 14:22:58.811423 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.811319 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a868a-predictor-serving-cert\"" May 20 14:22:58.836689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.836668 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:22:58.838602 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.838576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.838700 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.838624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6jn\" (UniqueName: \"kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.838783 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.838762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.939543 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.939511 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.939693 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.939554 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6jn\" (UniqueName: \"kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.939693 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.939616 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.939852 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:22:58.939738 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a868a-predictor-serving-cert: secret "success-200-isvc-a868a-predictor-serving-cert" not found May 20 14:22:58.939852 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:22:58.939840 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls podName:b7e32b31-f57b-4299-a06e-ae39800bbbdf nodeName:}" failed. No retries permitted until 2026-05-20 14:22:59.439817036 +0000 UTC m=+1964.190629411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls") pod "success-200-isvc-a868a-predictor-7675b7748-5r6xk" (UID: "b7e32b31-f57b-4299-a06e-ae39800bbbdf") : secret "success-200-isvc-a868a-predictor-serving-cert" not found May 20 14:22:58.940131 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.940108 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:58.950360 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:58.950333 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6jn\" (UniqueName: \"kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:59.443282 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.443248 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:59.445679 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.445656 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") pod \"success-200-isvc-a868a-predictor-7675b7748-5r6xk\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:59.643070 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.643032 2575 generic.go:358] "Generic (PLEG): container finished" podID="6a1303af-b906-402f-9ba8-44501594a4b4" containerID="13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6" exitCode=2 May 20 14:22:59.643231 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.643107 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerDied","Data":"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6"} May 20 14:22:59.719783 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.719680 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:22:59.853370 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.853320 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:22:59.855529 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:22:59.855506 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e32b31_f57b_4299_a06e_ae39800bbbdf.slice/crio-833d51d8cdf3a1887dee050dc69b74c84ea6ccf2251381fbf88d1c0c0a4b66d6 WatchSource:0}: Error finding container 833d51d8cdf3a1887dee050dc69b74c84ea6ccf2251381fbf88d1c0c0a4b66d6: Status 404 returned error can't find the container with id 833d51d8cdf3a1887dee050dc69b74c84ea6ccf2251381fbf88d1c0c0a4b66d6 May 20 14:22:59.857153 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:22:59.857138 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:23:00.648432 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.648385 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerStarted","Data":"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8"} May 20 14:23:00.648432 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.648429 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerStarted","Data":"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a"} May 20 14:23:00.648694 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.648444 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerStarted","Data":"833d51d8cdf3a1887dee050dc69b74c84ea6ccf2251381fbf88d1c0c0a4b66d6"} May 20 14:23:00.648694 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.648464 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:23:00.648694 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.648499 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:23:00.650049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.650022 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:00.668710 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:00.668604 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podStartSLOduration=2.66859031 podStartE2EDuration="2.66859031s" podCreationTimestamp="2026-05-20 14:22:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:23:00.668114785 +0000 UTC m=+1965.418927185" watchObservedRunningTime="2026-05-20 14:23:00.66859031 +0000 UTC m=+1965.419402709" May 20 14:23:01.102803 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.102695 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused" May 20 14:23:01.107055 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.107032 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" May 20 14:23:01.652038 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.652006 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:01.799803 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.799782 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:23:01.867549 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.867521 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config\") pod \"6a1303af-b906-402f-9ba8-44501594a4b4\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " May 20 14:23:01.867689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.867598 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhx2q\" (UniqueName: \"kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q\") pod \"6a1303af-b906-402f-9ba8-44501594a4b4\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " May 20 14:23:01.867689 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.867669 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls\") pod \"6a1303af-b906-402f-9ba8-44501594a4b4\" (UID: \"6a1303af-b906-402f-9ba8-44501594a4b4\") " May 20 14:23:01.867783 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.867720 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-ecafe-kube-rbac-proxy-sar-config") pod "6a1303af-b906-402f-9ba8-44501594a4b4" (UID: "6a1303af-b906-402f-9ba8-44501594a4b4"). InnerVolumeSpecName "success-200-isvc-ecafe-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:23:01.867857 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.867842 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-ecafe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1303af-b906-402f-9ba8-44501594a4b4-success-200-isvc-ecafe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:01.869569 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.869539 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a1303af-b906-402f-9ba8-44501594a4b4" (UID: "6a1303af-b906-402f-9ba8-44501594a4b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:23:01.869679 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.869570 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q" (OuterVolumeSpecName: "kube-api-access-zhx2q") pod "6a1303af-b906-402f-9ba8-44501594a4b4" (UID: "6a1303af-b906-402f-9ba8-44501594a4b4"). InnerVolumeSpecName "kube-api-access-zhx2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:23:01.969091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.969004 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1303af-b906-402f-9ba8-44501594a4b4-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:01.969091 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:01.969031 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhx2q\" (UniqueName: \"kubernetes.io/projected/6a1303af-b906-402f-9ba8-44501594a4b4-kube-api-access-zhx2q\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:02.655170 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.655132 2575 generic.go:358] "Generic (PLEG): container finished" podID="6a1303af-b906-402f-9ba8-44501594a4b4" containerID="2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e" exitCode=0 May 20 14:23:02.655623 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.655218 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" May 20 14:23:02.655623 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.655216 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerDied","Data":"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e"} May 20 14:23:02.655623 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.655329 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq" event={"ID":"6a1303af-b906-402f-9ba8-44501594a4b4","Type":"ContainerDied","Data":"4adcc2dafb67705997e4fd6421cf8ffd7584e933703e98100d106110a03bb7d1"} May 20 14:23:02.655623 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.655344 2575 scope.go:117] "RemoveContainer" containerID="13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6" May 20 14:23:02.663495 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.663474 2575 scope.go:117] "RemoveContainer" containerID="2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e" May 20 14:23:02.670200 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.670180 2575 scope.go:117] "RemoveContainer" containerID="13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6" May 20 14:23:02.670415 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:23:02.670398 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6\": container with ID starting with 13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6 not found: ID does not exist" containerID="13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6" May 20 14:23:02.670463 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.670425 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6"} err="failed to get container status \"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6\": rpc error: code = NotFound desc = could not find container \"13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6\": container with ID starting with 13517eaa2d4b8031b32d80bb90a40f6ff6223ee04ce5853d60e1dade0d87c3e6 not found: ID does not exist" May 20 14:23:02.670463 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.670442 2575 scope.go:117] "RemoveContainer" containerID="2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e" May 20 14:23:02.670678 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:23:02.670661 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e\": container with ID starting with 2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e not found: ID does not exist" containerID="2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e" May 20 14:23:02.670727 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.670683 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e"} err="failed to get container status \"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e\": rpc error: code = NotFound desc = could not find container \"2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e\": container with ID starting with 2c13d6b82d94eeb05e4857eeddd35c1e3bf9d25bb125a1ed9f1960893742730e not found: ID does not exist" May 20 14:23:02.679594 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.679574 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:23:02.685315 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:02.685297 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ecafe-predictor-785fb74b88-nx9dq"] May 20 14:23:03.841886 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:03.841854 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" path="/var/lib/kubelet/pods/6a1303af-b906-402f-9ba8-44501594a4b4/volumes" May 20 14:23:06.655836 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:06.655755 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:23:06.656230 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:06.656185 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:16.656226 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:16.656185 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:26.656755 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:26.656711 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:28.596005 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.595974 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:23:28.596407 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.596250 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" containerID="cri-o://15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb" gracePeriod=30 May 20 14:23:28.596407 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.596306 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kube-rbac-proxy" containerID="cri-o://c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48" gracePeriod=30 May 20 14:23:28.630594 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630569 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:23:28.630892 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" May 20 14:23:28.630934 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630894 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" May 20 14:23:28.630934 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630911 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kube-rbac-proxy" May 20 14:23:28.630934 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630918 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kube-rbac-proxy" May 20 14:23:28.631032 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630959 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kserve-container" May 20 14:23:28.631032 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.630968 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a1303af-b906-402f-9ba8-44501594a4b4" containerName="kube-rbac-proxy" May 20 14:23:28.633952 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.633937 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.639176 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.639154 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e303-predictor-serving-cert\"" May 20 14:23:28.639276 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.639231 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e303-kube-rbac-proxy-sar-config\"" May 20 14:23:28.653435 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.653407 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:23:28.735092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.735029 2575 generic.go:358] "Generic (PLEG): container finished" podID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerID="c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48" exitCode=2 May 20 14:23:28.735092 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.735072 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerDied","Data":"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48"} May 20 14:23:28.790976 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.790949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7mw\" (UniqueName: \"kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.791126 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.790988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.791126 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.791060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.891955 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.891915 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.892122 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.891977 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7mw\" (UniqueName: \"kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.892122 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.892004 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.892739 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.892716 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.894415 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.894393 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.901273 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.901254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7mw\" (UniqueName: \"kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw\") pod \"success-200-isvc-3e303-predictor-54ff64d9f6-79pgj\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:28.943085 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:28.943062 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:29.071176 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.071152 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:23:29.073619 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:23:29.073594 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1008ccb_006b_427d_974d_c62139642892.slice/crio-839acf12e498d4cf9c3c19beef2f23774e386f9fb82d283044c709b7f76e517b WatchSource:0}: Error finding container 839acf12e498d4cf9c3c19beef2f23774e386f9fb82d283044c709b7f76e517b: Status 404 returned error can't find the container with id 839acf12e498d4cf9c3c19beef2f23774e386f9fb82d283044c709b7f76e517b May 20 14:23:29.739972 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.739938 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerStarted","Data":"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011"} May 20 14:23:29.739972 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.739975 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerStarted","Data":"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be"} May 20 14:23:29.740408 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.739986 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerStarted","Data":"839acf12e498d4cf9c3c19beef2f23774e386f9fb82d283044c709b7f76e517b"} May 20 14:23:29.740408 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.740167 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:29.740408 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.740191 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:29.741476 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.741448 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:23:29.761693 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:29.761651 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podStartSLOduration=1.761638052 podStartE2EDuration="1.761638052s" podCreationTimestamp="2026-05-20 14:23:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:23:29.759876469 +0000 UTC m=+1994.510688865" watchObservedRunningTime="2026-05-20 14:23:29.761638052 +0000 UTC m=+1994.512450449" May 20 14:23:30.742681 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:30.742643 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:23:31.540942 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.540921 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:23:31.613120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.613054 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") pod \"d70962d5-e2e1-45e6-b610-376b13dc6648\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " May 20 14:23:31.613120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.613102 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config\") pod \"d70962d5-e2e1-45e6-b610-376b13dc6648\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " May 20 14:23:31.613277 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.613144 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf6ln\" (UniqueName: \"kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln\") pod \"d70962d5-e2e1-45e6-b610-376b13dc6648\" (UID: \"d70962d5-e2e1-45e6-b610-376b13dc6648\") " May 20 14:23:31.613531 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.613501 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-734fc-kube-rbac-proxy-sar-config") pod "d70962d5-e2e1-45e6-b610-376b13dc6648" (UID: "d70962d5-e2e1-45e6-b610-376b13dc6648"). InnerVolumeSpecName "success-200-isvc-734fc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:23:31.615032 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.615011 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d70962d5-e2e1-45e6-b610-376b13dc6648" (UID: "d70962d5-e2e1-45e6-b610-376b13dc6648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:23:31.615140 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.615121 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln" (OuterVolumeSpecName: "kube-api-access-gf6ln") pod "d70962d5-e2e1-45e6-b610-376b13dc6648" (UID: "d70962d5-e2e1-45e6-b610-376b13dc6648"). InnerVolumeSpecName "kube-api-access-gf6ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:23:31.714284 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.714253 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf6ln\" (UniqueName: \"kubernetes.io/projected/d70962d5-e2e1-45e6-b610-376b13dc6648-kube-api-access-gf6ln\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:31.714284 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.714280 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70962d5-e2e1-45e6-b610-376b13dc6648-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:31.714284 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.714290 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-734fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70962d5-e2e1-45e6-b610-376b13dc6648-success-200-isvc-734fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:23:31.747214 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.747178 2575 generic.go:358] "Generic (PLEG): container finished" podID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerID="15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb" exitCode=0 May 20 14:23:31.747586 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.747235 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerDied","Data":"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb"} May 20 14:23:31.747586 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.747271 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" event={"ID":"d70962d5-e2e1-45e6-b610-376b13dc6648","Type":"ContainerDied","Data":"d6cdd83bec9edca48b98e2e6ad44f6b9125b7506d4e0b1b3b27e83f3b58c4ecc"} May 20 14:23:31.747586 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.747278 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg" May 20 14:23:31.747586 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.747293 2575 scope.go:117] "RemoveContainer" containerID="c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48" May 20 14:23:31.756086 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.756069 2575 scope.go:117] "RemoveContainer" containerID="15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb" May 20 14:23:31.763096 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.763081 2575 scope.go:117] "RemoveContainer" containerID="c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48" May 20 14:23:31.763327 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:23:31.763305 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48\": container with ID starting with c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48 not found: ID does not exist" containerID="c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48" May 20 14:23:31.763375 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.763335 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48"} err="failed to get container status \"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48\": rpc error: code = NotFound desc = could not find container \"c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48\": container with ID starting with c5cea964dfeb9c0e01f83d9a3b19dc624e16b27876737b34af6341c30b010b48 not found: ID does not exist" May 20 14:23:31.763375 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.763357 2575 scope.go:117] "RemoveContainer" containerID="15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb" May 20 14:23:31.763571 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:23:31.763552 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb\": container with ID starting with 15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb not found: ID does not exist" containerID="15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb" May 20 14:23:31.763626 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.763581 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb"} err="failed to get container status \"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb\": rpc error: code = NotFound desc = could not find container \"15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb\": container with ID starting with 15d3dcee89226d4c6af74429843515553fe77a9eb1cb0c10fe9642e4605e26eb not found: ID does not exist" May 20 14:23:31.772673 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.772653 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:23:31.778021 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.778001 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-734fc-predictor-64d98dd8b6-bknhg"] May 20 14:23:31.842041 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:31.842017 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" path="/var/lib/kubelet/pods/d70962d5-e2e1-45e6-b610-376b13dc6648/volumes" May 20 14:23:35.748156 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:35.748123 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:23:35.748682 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:35.748652 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:23:36.656808 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:36.656775 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" May 20 14:23:45.749599 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:45.749558 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:23:46.656937 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:46.656902 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:23:55.749042 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:23:55.748999 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:24:05.749295 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:05.749251 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" May 20 14:24:09.080274 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.080243 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:24:09.080645 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.080505 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" containerID="cri-o://433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a" gracePeriod=30 May 20 14:24:09.080645 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.080572 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kube-rbac-proxy" containerID="cri-o://46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8" gracePeriod=30 May 20 14:24:09.103833 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.103802 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:24:09.104107 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104095 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" May 20 14:24:09.104151 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104108 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" May 20 14:24:09.104151 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104125 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kube-rbac-proxy" May 20 14:24:09.104151 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104131 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kube-rbac-proxy" May 20 14:24:09.104242 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104200 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kube-rbac-proxy" May 20 14:24:09.104242 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.104211 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="d70962d5-e2e1-45e6-b610-376b13dc6648" containerName="kserve-container" May 20 14:24:09.107327 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.107310 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.110067 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.110046 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\"" May 20 14:24:09.110324 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.110308 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31b2e-predictor-serving-cert\"" May 20 14:24:09.117720 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.117699 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:24:09.213335 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.213299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5xm\" (UniqueName: \"kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.213506 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.213356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.213506 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.213463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.314331 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.314296 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.314496 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.314449 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.314558 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.314528 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5xm\" (UniqueName: \"kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.314639 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:24:09.314620 2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-serving-cert: secret "success-200-isvc-31b2e-predictor-serving-cert" not found May 20 14:24:09.314694 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:24:09.314685 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls podName:1a64186b-ecb2-4088-9b56-24830d81d178 nodeName:}" failed. No retries permitted until 2026-05-20 14:24:09.814668966 +0000 UTC m=+2034.565481341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls") pod "success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" (UID: "1a64186b-ecb2-4088-9b56-24830d81d178") : secret "success-200-isvc-31b2e-predictor-serving-cert" not found May 20 14:24:09.314979 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.314960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.333351 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.333294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5xm\" (UniqueName: \"kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.818098 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.818065 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.820398 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.820369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") pod \"success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:09.856674 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.856642 2575 generic.go:358] "Generic (PLEG): container finished" podID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerID="46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8" exitCode=2 May 20 14:24:09.856833 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:09.856673 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerDied","Data":"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8"} May 20 14:24:10.017614 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.017567 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:10.155576 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.155314 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:24:10.160501 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:24:10.160474 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a64186b_ecb2_4088_9b56_24830d81d178.slice/crio-05f8e5dc5fb4d386b2a0e524152efd127bdceb6ab60e885d015c8bffa0290eb2 WatchSource:0}: Error finding container 05f8e5dc5fb4d386b2a0e524152efd127bdceb6ab60e885d015c8bffa0290eb2: Status 404 returned error can't find the container with id 05f8e5dc5fb4d386b2a0e524152efd127bdceb6ab60e885d015c8bffa0290eb2 May 20 14:24:10.861803 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.861762 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerStarted","Data":"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d"} May 20 14:24:10.861803 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.861804 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerStarted","Data":"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da"} May 20 14:24:10.862197 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.861814 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerStarted","Data":"05f8e5dc5fb4d386b2a0e524152efd127bdceb6ab60e885d015c8bffa0290eb2"} May 20 14:24:10.862197 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.861915 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:10.883844 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:10.883785 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podStartSLOduration=1.883772085 podStartE2EDuration="1.883772085s" podCreationTimestamp="2026-05-20 14:24:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:10.882299688 +0000 UTC m=+2035.633112088" watchObservedRunningTime="2026-05-20 14:24:10.883772085 +0000 UTC m=+2035.634584480" May 20 14:24:11.653068 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:11.653025 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused" May 20 14:24:11.865055 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:11.865022 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:11.866398 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:11.866368 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:12.622272 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.622252 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:24:12.744638 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.744565 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w6jn\" (UniqueName: \"kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn\") pod \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " May 20 14:24:12.745062 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.744637 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") pod \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " May 20 14:24:12.745062 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.744708 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config\") pod \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\" (UID: \"b7e32b31-f57b-4299-a06e-ae39800bbbdf\") " May 20 14:24:12.745268 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.745239 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-a868a-kube-rbac-proxy-sar-config") pod "b7e32b31-f57b-4299-a06e-ae39800bbbdf" (UID: "b7e32b31-f57b-4299-a06e-ae39800bbbdf"). InnerVolumeSpecName "success-200-isvc-a868a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:24:12.746719 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.746687 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7e32b31-f57b-4299-a06e-ae39800bbbdf" (UID: "b7e32b31-f57b-4299-a06e-ae39800bbbdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:24:12.746861 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.746762 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn" (OuterVolumeSpecName: "kube-api-access-8w6jn") pod "b7e32b31-f57b-4299-a06e-ae39800bbbdf" (UID: "b7e32b31-f57b-4299-a06e-ae39800bbbdf"). InnerVolumeSpecName "kube-api-access-8w6jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:24:12.845790 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.845733 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e32b31-f57b-4299-a06e-ae39800bbbdf-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:24:12.845790 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.845791 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-a868a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7e32b31-f57b-4299-a06e-ae39800bbbdf-success-200-isvc-a868a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:24:12.845965 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.845809 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w6jn\" (UniqueName: \"kubernetes.io/projected/b7e32b31-f57b-4299-a06e-ae39800bbbdf-kube-api-access-8w6jn\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:24:12.869126 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869100 2575 generic.go:358] "Generic (PLEG): container finished" podID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerID="433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a" exitCode=0 May 20 14:24:12.869260 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869183 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" May 20 14:24:12.869260 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869187 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerDied","Data":"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a"} May 20 14:24:12.869260 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869221 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk" event={"ID":"b7e32b31-f57b-4299-a06e-ae39800bbbdf","Type":"ContainerDied","Data":"833d51d8cdf3a1887dee050dc69b74c84ea6ccf2251381fbf88d1c0c0a4b66d6"} May 20 14:24:12.869260 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869238 2575 scope.go:117] "RemoveContainer" containerID="46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8" May 20 14:24:12.869820 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.869795 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:12.877361 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.877341 2575 scope.go:117] "RemoveContainer" containerID="433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a" May 20 14:24:12.883959 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.883943 2575 scope.go:117] "RemoveContainer" containerID="46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8" May 20 14:24:12.884200 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:24:12.884179 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8\": container with ID starting with 46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8 not found: ID does not exist" containerID="46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8" May 20 14:24:12.884242 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.884208 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8"} err="failed to get container status \"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8\": rpc error: code = NotFound desc = could not find container \"46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8\": container with ID starting with 46be0bbadb7626c709651adb9b8c131bf382a3f1aaeb99517e631b3a94f769e8 not found: ID does not exist" May 20 14:24:12.884242 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.884226 2575 scope.go:117] "RemoveContainer" containerID="433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a" May 20 14:24:12.884437 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:24:12.884420 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a\": container with ID starting with 433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a not found: ID does not exist" containerID="433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a" May 20 14:24:12.884471 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.884443 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a"} err="failed to get container status \"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a\": rpc error: code = NotFound desc = could not find container \"433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a\": container with ID starting with 433660250adcdc2e3921e0072e9035fc03b024b54b02d44e215aa8040476338a not found: ID does not exist" May 20 14:24:12.892055 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.892036 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:24:12.896964 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:12.896944 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a868a-predictor-7675b7748-5r6xk"] May 20 14:24:13.842624 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:13.842593 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" path="/var/lib/kubelet/pods/b7e32b31-f57b-4299-a06e-ae39800bbbdf/volumes" May 20 14:24:15.749502 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:15.749473 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:24:17.873859 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:17.873831 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:24:17.874343 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:17.874319 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:27.875310 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:27.875276 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:37.874616 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:37.874535 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:47.874560 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:47.874518 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" May 20 14:24:57.874871 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:24:57.874842 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:33:23.940894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:23.940855 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:33:23.943259 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:23.941130 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" containerID="cri-o://cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da" gracePeriod=30 May 20 14:33:23.943259 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:23.941173 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kube-rbac-proxy" containerID="cri-o://ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d" gracePeriod=30 May 20 14:33:24.420331 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:24.420297 2575 generic.go:358] "Generic (PLEG): container finished" podID="1a64186b-ecb2-4088-9b56-24830d81d178" containerID="ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d" exitCode=2 May 20 14:33:24.420501 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:24.420364 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerDied","Data":"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d"} May 20 14:33:27.284887 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.284856 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:33:27.369638 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.369607 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t5xm\" (UniqueName: \"kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm\") pod \"1a64186b-ecb2-4088-9b56-24830d81d178\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " May 20 14:33:27.369815 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.369667 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") pod \"1a64186b-ecb2-4088-9b56-24830d81d178\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " May 20 14:33:27.369815 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.369702 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config\") pod \"1a64186b-ecb2-4088-9b56-24830d81d178\" (UID: \"1a64186b-ecb2-4088-9b56-24830d81d178\") " May 20 14:33:27.370120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.370093 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-31b2e-kube-rbac-proxy-sar-config") pod "1a64186b-ecb2-4088-9b56-24830d81d178" (UID: "1a64186b-ecb2-4088-9b56-24830d81d178"). InnerVolumeSpecName "success-200-isvc-31b2e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:33:27.371726 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.371702 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a64186b-ecb2-4088-9b56-24830d81d178" (UID: "1a64186b-ecb2-4088-9b56-24830d81d178"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:33:27.371862 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.371785 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm" (OuterVolumeSpecName: "kube-api-access-4t5xm") pod "1a64186b-ecb2-4088-9b56-24830d81d178" (UID: "1a64186b-ecb2-4088-9b56-24830d81d178"). InnerVolumeSpecName "kube-api-access-4t5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:33:27.430821 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.430787 2575 generic.go:358] "Generic (PLEG): container finished" podID="1a64186b-ecb2-4088-9b56-24830d81d178" containerID="cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da" exitCode=0 May 20 14:33:27.430984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.430856 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerDied","Data":"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da"} May 20 14:33:27.430984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.430897 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" May 20 14:33:27.430984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.430906 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx" event={"ID":"1a64186b-ecb2-4088-9b56-24830d81d178","Type":"ContainerDied","Data":"05f8e5dc5fb4d386b2a0e524152efd127bdceb6ab60e885d015c8bffa0290eb2"} May 20 14:33:27.430984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.430927 2575 scope.go:117] "RemoveContainer" containerID="ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d" May 20 14:33:27.438762 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.438721 2575 scope.go:117] "RemoveContainer" containerID="cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da" May 20 14:33:27.448192 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.448169 2575 scope.go:117] "RemoveContainer" containerID="ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d" May 20 14:33:27.448488 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:33:27.448471 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d\": container with ID starting with ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d not found: ID does not exist" containerID="ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d" May 20 14:33:27.448533 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.448497 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d"} err="failed to get container status \"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d\": rpc error: code = NotFound desc = could not find container \"ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d\": container with ID starting with ff969e702c320e3d8a2001efb31ce346795fec4c2a40a174e87a9f524b7f8f5d not found: ID does not exist" May 20 14:33:27.448533 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.448513 2575 scope.go:117] "RemoveContainer" containerID="cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da" May 20 14:33:27.448786 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:33:27.448737 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da\": container with ID starting with cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da not found: ID does not exist" containerID="cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da" May 20 14:33:27.448786 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.448787 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da"} err="failed to get container status \"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da\": rpc error: code = NotFound desc = could not find container \"cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da\": container with ID starting with cd3252842ddb20910f181255a3792fcb28493bc6afc76477b1c6c52b5dfae3da not found: ID does not exist" May 20 14:33:27.459694 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.459668 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:33:27.466514 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.466492 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31b2e-predictor-67cf4677b6-p2vdx"] May 20 14:33:27.470948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.470922 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a64186b-ecb2-4088-9b56-24830d81d178-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:33:27.470948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.470946 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-31b2e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a64186b-ecb2-4088-9b56-24830d81d178-success-200-isvc-31b2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:33:27.471080 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.470958 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5xm\" (UniqueName: \"kubernetes.io/projected/1a64186b-ecb2-4088-9b56-24830d81d178-kube-api-access-4t5xm\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:33:27.842578 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:33:27.842499 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" path="/var/lib/kubelet/pods/1a64186b-ecb2-4088-9b56-24830d81d178/volumes" May 20 14:40:48.081407 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:48.081375 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:40:48.083976 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:48.081664 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" containerID="cri-o://05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be" gracePeriod=30 May 20 14:40:48.083976 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:48.081706 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kube-rbac-proxy" containerID="cri-o://3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011" gracePeriod=30 May 20 14:40:48.638204 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:48.638175 2575 generic.go:358] "Generic (PLEG): container finished" podID="e1008ccb-006b-427d-974d-c62139642892" containerID="3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011" exitCode=2 May 20 14:40:48.638365 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:48.638248 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerDied","Data":"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011"} May 20 14:40:49.349587 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.349558 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-22sst/must-gather-scbvq"] May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350005 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350025 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350045 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kube-rbac-proxy" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350052 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kube-rbac-proxy" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350064 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kube-rbac-proxy" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350072 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kube-rbac-proxy" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350094 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" May 20 14:40:49.350104 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350102 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" May 20 14:40:49.350502 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350168 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kube-rbac-proxy" May 20 14:40:49.350502 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350186 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7e32b31-f57b-4299-a06e-ae39800bbbdf" containerName="kserve-container" May 20 14:40:49.350502 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350195 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kube-rbac-proxy" May 20 14:40:49.350502 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.350207 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a64186b-ecb2-4088-9b56-24830d81d178" containerName="kserve-container" May 20 14:40:49.353133 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.353113 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.355497 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.355476 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22sst\"/\"kube-root-ca.crt\"" May 20 14:40:49.355616 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.355478 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-22sst\"/\"default-dockercfg-dwmt9\"" May 20 14:40:49.355616 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.355518 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22sst\"/\"openshift-service-ca.crt\"" May 20 14:40:49.367310 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.367281 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22sst/must-gather-scbvq"] May 20 14:40:49.447546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.447510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26fl\" (UniqueName: \"kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.447780 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.447602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.548345 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.548317 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.548465 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.548367 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r26fl\" (UniqueName: \"kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.548677 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.548654 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.558005 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.557977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26fl\" (UniqueName: \"kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl\") pod \"must-gather-scbvq\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.678779 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.678734 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:40:49.800351 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.800323 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22sst/must-gather-scbvq"] May 20 14:40:49.803698 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:40:49.803667 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aedd6f9_cf33_439b_9d4e_503dd2d36d05.slice/crio-ee9d653ac247149381785f075b23181ca2297071df73f28cd7eec69cc9e819b2 WatchSource:0}: Error finding container ee9d653ac247149381785f075b23181ca2297071df73f28cd7eec69cc9e819b2: Status 404 returned error can't find the container with id ee9d653ac247149381785f075b23181ca2297071df73f28cd7eec69cc9e819b2 May 20 14:40:49.805303 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:49.805286 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:40:50.648807 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:50.648727 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22sst/must-gather-scbvq" event={"ID":"2aedd6f9-cf33-439b-9d4e-503dd2d36d05","Type":"ContainerStarted","Data":"ee9d653ac247149381785f075b23181ca2297071df73f28cd7eec69cc9e819b2"} May 20 14:40:50.744185 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:50.744141 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused" May 20 14:40:51.173122 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.172866 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:40:51.264090 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.263818 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7mw\" (UniqueName: \"kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw\") pod \"e1008ccb-006b-427d-974d-c62139642892\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " May 20 14:40:51.264090 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.263914 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls\") pod \"e1008ccb-006b-427d-974d-c62139642892\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " May 20 14:40:51.264090 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.263986 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config\") pod \"e1008ccb-006b-427d-974d-c62139642892\" (UID: \"e1008ccb-006b-427d-974d-c62139642892\") " May 20 14:40:51.264515 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.264475 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-3e303-kube-rbac-proxy-sar-config") pod "e1008ccb-006b-427d-974d-c62139642892" (UID: "e1008ccb-006b-427d-974d-c62139642892"). InnerVolumeSpecName "success-200-isvc-3e303-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:40:51.266551 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.266526 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1008ccb-006b-427d-974d-c62139642892" (UID: "e1008ccb-006b-427d-974d-c62139642892"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:40:51.267930 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.267895 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw" (OuterVolumeSpecName: "kube-api-access-2z7mw") pod "e1008ccb-006b-427d-974d-c62139642892" (UID: "e1008ccb-006b-427d-974d-c62139642892"). InnerVolumeSpecName "kube-api-access-2z7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:40:51.365460 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.365424 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7mw\" (UniqueName: \"kubernetes.io/projected/e1008ccb-006b-427d-974d-c62139642892-kube-api-access-2z7mw\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:40:51.365460 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.365459 2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1008ccb-006b-427d-974d-c62139642892-proxy-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:40:51.365667 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.365476 2575 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-3e303-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1008ccb-006b-427d-974d-c62139642892-success-200-isvc-3e303-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:40:51.655023 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.654972 2575 generic.go:358] "Generic (PLEG): container finished" podID="e1008ccb-006b-427d-974d-c62139642892" containerID="05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be" exitCode=0 May 20 14:40:51.655023 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.655034 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerDied","Data":"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be"} May 20 14:40:51.655546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.655064 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" event={"ID":"e1008ccb-006b-427d-974d-c62139642892","Type":"ContainerDied","Data":"839acf12e498d4cf9c3c19beef2f23774e386f9fb82d283044c709b7f76e517b"} May 20 14:40:51.655546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.655078 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj" May 20 14:40:51.655546 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.655082 2575 scope.go:117] "RemoveContainer" containerID="3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011" May 20 14:40:51.666797 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.666672 2575 scope.go:117] "RemoveContainer" containerID="05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be" May 20 14:40:51.677554 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.677526 2575 scope.go:117] "RemoveContainer" containerID="3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011" May 20 14:40:51.677883 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:40:51.677856 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011\": container with ID starting with 3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011 not found: ID does not exist" containerID="3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011" May 20 14:40:51.677971 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.677895 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011"} err="failed to get container status \"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011\": rpc error: code = NotFound desc = could not find container \"3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011\": container with ID starting with 3f5448185f06692fc1c7980a50a42036502fde3fc2c6a3ad874d49db7b1d7011 not found: ID does not exist" May 20 14:40:51.677971 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.677920 2575 scope.go:117] "RemoveContainer" containerID="05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be" May 20 14:40:51.678699 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:40:51.678671 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be\": container with ID starting with 05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be not found: ID does not exist" containerID="05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be" May 20 14:40:51.678808 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.678706 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be"} err="failed to get container status \"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be\": rpc error: code = NotFound desc = could not find container \"05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be\": container with ID starting with 05de30c81d3026b4be9002461a13e06a22dea75a5f59a2b90bdde789732fc1be not found: ID does not exist" May 20 14:40:51.689387 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.689362 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:40:51.695641 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.695617 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e303-predictor-54ff64d9f6-79pgj"] May 20 14:40:51.847158 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:51.847121 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1008ccb-006b-427d-974d-c62139642892" path="/var/lib/kubelet/pods/e1008ccb-006b-427d-974d-c62139642892/volumes" May 20 14:40:56.672106 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:56.672071 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22sst/must-gather-scbvq" event={"ID":"2aedd6f9-cf33-439b-9d4e-503dd2d36d05","Type":"ContainerStarted","Data":"499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4"} May 20 14:40:56.672106 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:56.672107 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22sst/must-gather-scbvq" event={"ID":"2aedd6f9-cf33-439b-9d4e-503dd2d36d05","Type":"ContainerStarted","Data":"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27"} May 20 14:40:56.689789 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:40:56.689715 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-22sst/must-gather-scbvq" podStartSLOduration=1.645621755 podStartE2EDuration="7.689671791s" podCreationTimestamp="2026-05-20 14:40:49 +0000 UTC" firstStartedPulling="2026-05-20 14:40:49.805430467 +0000 UTC m=+3034.556242841" lastFinishedPulling="2026-05-20 14:40:55.849480502 +0000 UTC m=+3040.600292877" observedRunningTime="2026-05-20 14:40:56.688952734 +0000 UTC m=+3041.439765132" watchObservedRunningTime="2026-05-20 14:40:56.689671791 +0000 UTC m=+3041.440484188" May 20 14:41:18.738507 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:18.738474 2575 generic.go:358] "Generic (PLEG): container finished" podID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerID="8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27" exitCode=0 May 20 14:41:18.738947 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:18.738555 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22sst/must-gather-scbvq" event={"ID":"2aedd6f9-cf33-439b-9d4e-503dd2d36d05","Type":"ContainerDied","Data":"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27"} May 20 14:41:18.738947 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:18.738904 2575 scope.go:117] "RemoveContainer" containerID="8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27" May 20 14:41:18.805733 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:18.805693 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-22sst_must-gather-scbvq_2aedd6f9-cf33-439b-9d4e-503dd2d36d05/gather/0.log" May 20 14:41:19.421081 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421045 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvfml/must-gather-b46mv"] May 20 14:41:19.421367 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421356 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" May 20 14:41:19.421410 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421369 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" May 20 14:41:19.421410 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421385 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kube-rbac-proxy" May 20 14:41:19.421410 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421391 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kube-rbac-proxy" May 20 14:41:19.421501 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421437 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kserve-container" May 20 14:41:19.421501 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.421448 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1008ccb-006b-427d-974d-c62139642892" containerName="kube-rbac-proxy" May 20 14:41:19.423465 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.423450 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.426040 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.426013 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvfml\"/\"openshift-service-ca.crt\"" May 20 14:41:19.427019 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.427000 2575 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lvfml\"/\"default-dockercfg-54sx6\"" May 20 14:41:19.427117 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.427043 2575 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvfml\"/\"kube-root-ca.crt\"" May 20 14:41:19.432047 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.432022 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/must-gather-b46mv"] May 20 14:41:19.494253 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.494224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nbh\" (UniqueName: \"kubernetes.io/projected/58564c99-9f01-49f7-a86f-4975fea69d5f-kube-api-access-89nbh\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.494410 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.494259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/58564c99-9f01-49f7-a86f-4975fea69d5f-must-gather-output\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.594688 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.594660 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89nbh\" (UniqueName: \"kubernetes.io/projected/58564c99-9f01-49f7-a86f-4975fea69d5f-kube-api-access-89nbh\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.594958 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.594697 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/58564c99-9f01-49f7-a86f-4975fea69d5f-must-gather-output\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.595044 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.594965 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/58564c99-9f01-49f7-a86f-4975fea69d5f-must-gather-output\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.603376 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.603348 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nbh\" (UniqueName: \"kubernetes.io/projected/58564c99-9f01-49f7-a86f-4975fea69d5f-kube-api-access-89nbh\") pod \"must-gather-b46mv\" (UID: \"58564c99-9f01-49f7-a86f-4975fea69d5f\") " pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.732864 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.732798 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/must-gather-b46mv" May 20 14:41:19.857315 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:19.857190 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/must-gather-b46mv"] May 20 14:41:19.859717 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:41:19.859691 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58564c99_9f01_49f7_a86f_4975fea69d5f.slice/crio-778e08be9793fcdf2b7b58110ffb521c5f953435e9667f700a94f20467f29d54 WatchSource:0}: Error finding container 778e08be9793fcdf2b7b58110ffb521c5f953435e9667f700a94f20467f29d54: Status 404 returned error can't find the container with id 778e08be9793fcdf2b7b58110ffb521c5f953435e9667f700a94f20467f29d54 May 20 14:41:20.745928 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:20.745887 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/must-gather-b46mv" event={"ID":"58564c99-9f01-49f7-a86f-4975fea69d5f","Type":"ContainerStarted","Data":"540fbe09b27c65392a10228f665739852943ef33800a16c7956a77f0269ef274"} May 20 14:41:20.745928 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:20.745929 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/must-gather-b46mv" event={"ID":"58564c99-9f01-49f7-a86f-4975fea69d5f","Type":"ContainerStarted","Data":"778e08be9793fcdf2b7b58110ffb521c5f953435e9667f700a94f20467f29d54"} May 20 14:41:21.752411 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:21.752367 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/must-gather-b46mv" event={"ID":"58564c99-9f01-49f7-a86f-4975fea69d5f","Type":"ContainerStarted","Data":"5ef72b69a6f5bb142906793f21d9acb371a787afa385f9e53f6efa3288233c39"} May 20 14:41:22.265921 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:22.265895 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kdngv_376a9486-73e2-4fe4-b9a1-57419dc0c808/global-pull-secret-syncer/0.log" May 20 14:41:22.377307 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:22.377278 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lfcgh_fd716f72-80d8-4c8e-bfed-6f3053ad0a3b/konnectivity-agent/0.log" May 20 14:41:22.566581 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:22.566500 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-148.ec2.internal_18cf8ee325eb0038c1d0d88dc7ce7795/haproxy/0.log" May 20 14:41:24.273594 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.273531 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lvfml/must-gather-b46mv" podStartSLOduration=4.561310752 podStartE2EDuration="5.273511641s" podCreationTimestamp="2026-05-20 14:41:19 +0000 UTC" firstStartedPulling="2026-05-20 14:41:19.864023693 +0000 UTC m=+3064.614836082" lastFinishedPulling="2026-05-20 14:41:20.576224591 +0000 UTC m=+3065.327036971" observedRunningTime="2026-05-20 14:41:21.793382189 +0000 UTC m=+3066.544194587" watchObservedRunningTime="2026-05-20 14:41:24.273511641 +0000 UTC m=+3069.024324039" May 20 14:41:24.275009 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.274981 2575 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-22sst/must-gather-scbvq"] May 20 14:41:24.275464 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.275436 2575 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-22sst/must-gather-scbvq" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="copy" containerID="cri-o://499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4" gracePeriod=2 May 20 14:41:24.278299 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.278224 2575 status_manager.go:895] "Failed to get status for pod" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" pod="openshift-must-gather-22sst/must-gather-scbvq" err="pods \"must-gather-scbvq\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-22sst\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" May 20 14:41:24.280134 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.280109 2575 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-22sst/must-gather-scbvq"] May 20 14:41:24.650811 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.650720 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-22sst_must-gather-scbvq_2aedd6f9-cf33-439b-9d4e-503dd2d36d05/copy/0.log" May 20 14:41:24.651553 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.651288 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:41:24.653862 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.653830 2575 status_manager.go:895] "Failed to get status for pod" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" pod="openshift-must-gather-22sst/must-gather-scbvq" err="pods \"must-gather-scbvq\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-22sst\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" May 20 14:41:24.746449 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.745830 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output\") pod \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " May 20 14:41:24.746449 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.745901 2575 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r26fl\" (UniqueName: \"kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl\") pod \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\" (UID: \"2aedd6f9-cf33-439b-9d4e-503dd2d36d05\") " May 20 14:41:24.749825 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.749775 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl" (OuterVolumeSpecName: "kube-api-access-r26fl") pod "2aedd6f9-cf33-439b-9d4e-503dd2d36d05" (UID: "2aedd6f9-cf33-439b-9d4e-503dd2d36d05"). InnerVolumeSpecName "kube-api-access-r26fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:41:24.749959 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.749847 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "2aedd6f9-cf33-439b-9d4e-503dd2d36d05" (UID: "2aedd6f9-cf33-439b-9d4e-503dd2d36d05"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:41:24.765918 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.765847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-22sst_must-gather-scbvq_2aedd6f9-cf33-439b-9d4e-503dd2d36d05/copy/0.log" May 20 14:41:24.772778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.766420 2575 generic.go:358] "Generic (PLEG): container finished" podID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerID="499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4" exitCode=143 May 20 14:41:24.772778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.766525 2575 scope.go:117] "RemoveContainer" containerID="499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4" May 20 14:41:24.772778 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.766643 2575 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22sst/must-gather-scbvq" May 20 14:41:24.784138 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.777478 2575 status_manager.go:895] "Failed to get status for pod" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" pod="openshift-must-gather-22sst/must-gather-scbvq" err="pods \"must-gather-scbvq\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-22sst\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" May 20 14:41:24.792668 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.792625 2575 status_manager.go:895] "Failed to get status for pod" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" pod="openshift-must-gather-22sst/must-gather-scbvq" err="pods \"must-gather-scbvq\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-22sst\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" May 20 14:41:24.806899 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.805035 2575 scope.go:117] "RemoveContainer" containerID="8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27" May 20 14:41:24.830984 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.830953 2575 scope.go:117] "RemoveContainer" containerID="499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4" May 20 14:41:24.832191 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:41:24.832140 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4\": container with ID starting with 499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4 not found: ID does not exist" containerID="499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4" May 20 14:41:24.832408 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.832340 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4"} err="failed to get container status \"499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4\": rpc error: code = NotFound desc = could not find container \"499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4\": container with ID starting with 499beafcc6f429804623426261d29e67a95cd556ee077ff7a4e35061109684f4 not found: ID does not exist" May 20 14:41:24.832408 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.832371 2575 scope.go:117] "RemoveContainer" containerID="8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27" May 20 14:41:24.833171 ip-10-0-137-148 kubenswrapper[2575]: E0520 14:41:24.833112 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27\": container with ID starting with 8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27 not found: ID does not exist" containerID="8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27" May 20 14:41:24.833171 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.833140 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27"} err="failed to get container status \"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27\": rpc error: code = NotFound desc = could not find container \"8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27\": container with ID starting with 8d12965bb316e3c563ab1c1e66a0f184634b10fb752e02fcc530a421b12b1a27 not found: ID does not exist" May 20 14:41:24.847018 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.846597 2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-must-gather-output\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:41:24.847018 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:24.846626 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26fl\" (UniqueName: \"kubernetes.io/projected/2aedd6f9-cf33-439b-9d4e-503dd2d36d05-kube-api-access-r26fl\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" May 20 14:41:25.844362 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:25.844323 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" path="/var/lib/kubelet/pods/2aedd6f9-cf33-439b-9d4e-503dd2d36d05/volumes" May 20 14:41:25.845007 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:25.844833 2575 status_manager.go:895] "Failed to get status for pod" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" pod="openshift-must-gather-22sst/must-gather-scbvq" err="pods \"must-gather-scbvq\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-22sst\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" May 20 14:41:26.454120 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.454091 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5847d95bf5-t7sjg_184090c7-5c7a-43bf-a1d6-fbae57f9689c/metrics-server/0.log" May 20 14:41:26.704527 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.704440 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwglq_a489e994-139d-4b9d-9af8-5899071bd764/node-exporter/0.log" May 20 14:41:26.729659 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.729634 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwglq_a489e994-139d-4b9d-9af8-5899071bd764/kube-rbac-proxy/0.log" May 20 14:41:26.755766 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.755708 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwglq_a489e994-139d-4b9d-9af8-5899071bd764/init-textfile/0.log" May 20 14:41:26.803723 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.803690 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6jqgn_53128b76-9783-465e-8fe3-fb2b4a93a082/kube-rbac-proxy-main/0.log" May 20 14:41:26.836768 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.836680 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6jqgn_53128b76-9783-465e-8fe3-fb2b4a93a082/kube-rbac-proxy-self/0.log" May 20 14:41:26.871196 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.871165 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6jqgn_53128b76-9783-465e-8fe3-fb2b4a93a082/openshift-state-metrics/0.log" May 20 14:41:26.927183 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.927141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/prometheus/0.log" May 20 14:41:26.994306 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:26.994235 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/config-reloader/0.log" May 20 14:41:27.103666 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.103642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/thanos-sidecar/0.log" May 20 14:41:27.174283 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.174249 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/kube-rbac-proxy-web/0.log" May 20 14:41:27.209971 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.209947 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/kube-rbac-proxy/0.log" May 20 14:41:27.238169 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.238074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/kube-rbac-proxy-thanos/0.log" May 20 14:41:27.270531 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.270284 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29bc0749-203a-49a5-bae4-d6c362ad8cf3/init-config-reloader/0.log" May 20 14:41:27.365265 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.365231 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-sqwsl_c97bb658-6ebe-40b1-b38c-c975b30f2f4a/prometheus-operator-admission-webhook/0.log" May 20 14:41:27.491648 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.491612 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/thanos-query/0.log" May 20 14:41:27.560221 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.560139 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/kube-rbac-proxy-web/0.log" May 20 14:41:27.589627 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.589580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/kube-rbac-proxy/0.log" May 20 14:41:27.616812 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.616781 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/prom-label-proxy/0.log" May 20 14:41:27.643803 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.643731 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/kube-rbac-proxy-rules/0.log" May 20 14:41:27.668770 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:27.668733 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94554c77f-9mskt_cafb9b21-4fe9-4222-b5ec-36e750d3cf37/kube-rbac-proxy-metrics/0.log" May 20 14:41:28.966894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.966859 2575 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5"] May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967173 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="copy" May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967184 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="copy" May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967193 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="gather" May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="gather" May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967254 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="gather" May 20 14:41:28.967469 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.967264 2575 memory_manager.go:356] "RemoveStaleState removing state" podUID="2aedd6f9-cf33-439b-9d4e-503dd2d36d05" containerName="copy" May 20 14:41:28.970247 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.970218 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:28.979913 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.979886 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5"] May 20 14:41:28.984758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.984459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-lib-modules\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:28.984758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.984506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-podres\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:28.984758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.984540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-sys\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:28.984758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.984580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxnv\" (UniqueName: \"kubernetes.io/projected/82111fe4-96bf-44c7-b023-5907357604b6-kube-api-access-cvxnv\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:28.984758 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:28.984608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-proc\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.085869 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085819 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-lib-modules\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085882 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-podres\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085917 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-sys\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085960 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxnv\" (UniqueName: \"kubernetes.io/projected/82111fe4-96bf-44c7-b023-5907357604b6-kube-api-access-cvxnv\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085988 2575 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-proc\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.085994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-lib-modules\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.086011 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-podres\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086049 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.086052 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-proc\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.086318 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.086053 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82111fe4-96bf-44c7-b023-5907357604b6-sys\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.097264 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.096388 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxnv\" (UniqueName: \"kubernetes.io/projected/82111fe4-96bf-44c7-b023-5907357604b6-kube-api-access-cvxnv\") pod \"perf-node-gather-daemonset-p4zj5\" (UID: \"82111fe4-96bf-44c7-b023-5907357604b6\") " pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.282864 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.282766 2575 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.434467 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.434440 2575 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5"] May 20 14:41:29.437039 ip-10-0-137-148 kubenswrapper[2575]: W0520 14:41:29.437011 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82111fe4_96bf_44c7_b023_5907357604b6.slice/crio-68221f39058c0e1b9c38e1a0b9dd02dddd5fe7376fc97e734d2bf80c0ff08fa1 WatchSource:0}: Error finding container 68221f39058c0e1b9c38e1a0b9dd02dddd5fe7376fc97e734d2bf80c0ff08fa1: Status 404 returned error can't find the container with id 68221f39058c0e1b9c38e1a0b9dd02dddd5fe7376fc97e734d2bf80c0ff08fa1 May 20 14:41:29.468619 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.468596 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-7lq7p_d6c5ea91-4d2f-4888-8f1d-83c4d6d830a5/download-server/0.log" May 20 14:41:29.785786 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.785721 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" event={"ID":"82111fe4-96bf-44c7-b023-5907357604b6","Type":"ContainerStarted","Data":"8f438f5949f1b958eaa96ba0c1d65e87a53d9baf6d0d0efba5a37c6aaaee3d72"} May 20 14:41:29.785948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.785795 2575 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" event={"ID":"82111fe4-96bf-44c7-b023-5907357604b6","Type":"ContainerStarted","Data":"68221f39058c0e1b9c38e1a0b9dd02dddd5fe7376fc97e734d2bf80c0ff08fa1"} May 20 14:41:29.785948 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.785828 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:29.803993 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:29.803882 2575 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" podStartSLOduration=1.803863545 podStartE2EDuration="1.803863545s" podCreationTimestamp="2026-05-20 14:41:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:41:29.802622168 +0000 UTC m=+3074.553434565" watchObservedRunningTime="2026-05-20 14:41:29.803863545 +0000 UTC m=+3074.554675944" May 20 14:41:30.560196 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:30.560169 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4njk8_386a74e3-df07-42cc-adc5-a68db86ebaed/dns/0.log" May 20 14:41:30.583871 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:30.583843 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4njk8_386a74e3-df07-42cc-adc5-a68db86ebaed/kube-rbac-proxy/0.log" May 20 14:41:30.768943 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:30.768913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tzl92_e3c70d7c-144e-473d-9344-f2947744cada/dns-node-resolver/0.log" May 20 14:41:31.199241 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:31.199205 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58f866d876-77fkh_68932548-56a5-44ca-a92b-2ce59cdb1c1e/registry/0.log" May 20 14:41:31.283424 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:31.283396 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nknqs_25c12a44-9f17-44ff-b915-c31404a6871e/node-ca/0.log" May 20 14:41:32.383515 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:32.383489 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7rgmg_ed6df236-b3f8-4ea9-a6eb-61236e031275/serve-healthcheck-canary/0.log" May 20 14:41:32.911900 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:32.911874 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxgd7_0c2fbd69-f884-4c3b-99e3-77c25f4af8c7/kube-rbac-proxy/0.log" May 20 14:41:32.939891 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:32.939865 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxgd7_0c2fbd69-f884-4c3b-99e3-77c25f4af8c7/exporter/0.log" May 20 14:41:32.964035 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:32.964013 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxgd7_0c2fbd69-f884-4c3b-99e3-77c25f4af8c7/extractor/0.log" May 20 14:41:35.395608 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:35.395580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pp88x_ce74e211-4c38-44bc-9cda-96120d8ec124/s3-init/0.log" May 20 14:41:35.415198 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:35.415172 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pp88x_ce74e211-4c38-44bc-9cda-96120d8ec124/download-hf-model/0.log" May 20 14:41:35.801452 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:35.801363 2575 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lvfml/perf-node-gather-daemonset-p4zj5" May 20 14:41:41.544783 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.544712 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/kube-multus-additional-cni-plugins/0.log" May 20 14:41:41.574857 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.574825 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/egress-router-binary-copy/0.log" May 20 14:41:41.601596 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.601567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/cni-plugins/0.log" May 20 14:41:41.631282 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.631258 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/bond-cni-plugin/0.log" May 20 14:41:41.658894 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.658863 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/routeoverride-cni/0.log" May 20 14:41:41.687088 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.687058 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/whereabouts-cni-bincopy/0.log" May 20 14:41:41.712883 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.712858 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z6nqd_d9ca5fdc-47a4-4a7c-b0df-16857b6f9dfa/whereabouts-cni/0.log" May 20 14:41:41.959279 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:41.959251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xlk7j_821307da-511e-49b1-be2f-7c58b70de56d/kube-multus/0.log" May 20 14:41:42.039194 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:42.039163 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5pjpk_b8a9e50a-d38e-4123-9a51-fde818feb241/network-metrics-daemon/0.log" May 20 14:41:42.068995 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:42.068965 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5pjpk_b8a9e50a-d38e-4123-9a51-fde818feb241/kube-rbac-proxy/0.log" May 20 14:41:43.271084 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.271058 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/ovn-controller/0.log" May 20 14:41:43.307696 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.307672 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/ovn-acl-logging/0.log" May 20 14:41:43.326972 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.326945 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/kube-rbac-proxy-node/0.log" May 20 14:41:43.355857 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.355822 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/kube-rbac-proxy-ovn-metrics/0.log" May 20 14:41:43.379379 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.379353 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/northd/0.log" May 20 14:41:43.407474 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.407448 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/nbdb/0.log" May 20 14:41:43.435911 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.435883 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/sbdb/0.log" May 20 14:41:43.557183 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:43.557111 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lhfql_6e9f8fc8-87cf-49f2-82f7-4232a3698889/ovnkube-controller/0.log" May 20 14:41:44.912510 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:44.912480 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dlhw8_c4fb9a1a-e7e7-47f2-af50-76404e0aabfb/network-check-target-container/0.log" May 20 14:41:45.946354 ip-10-0-137-148 kubenswrapper[2575]: I0520 14:41:45.946328 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-75246_2a5587d1-e164-4009-995a-e55a34ad59ab/iptables-alerter/0.log"