May 20 13:47:27.498386 ip-10-0-140-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory May 20 13:47:27.498401 ip-10-0-140-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory May 20 13:47:27.498410 ip-10-0-140-251 systemd[1]: kubelet.service: Failed with result 'resources'. May 20 13:47:27.498715 ip-10-0-140-251 systemd[1]: Failed to start Kubernetes Kubelet. May 20 13:47:37.537217 ip-10-0-140-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. May 20 13:47:37.537234 ip-10-0-140-251 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b0bd2205e5d543bfa0be7e7d52cc1ac8 -- May 20 13:50:04.167185 ip-10-0-140-251 systemd[1]: Starting Kubernetes Kubelet... May 20 13:50:04.667941 ip-10-0-140-251 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:04.667941 ip-10-0-140-251 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 20 13:50:04.667941 ip-10-0-140-251 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:04.667941 ip-10-0-140-251 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 20 13:50:04.667941 ip-10-0-140-251 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 20 13:50:04.668781 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.668688 2554 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 20 13:50:04.672069 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672054 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:04.672069 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672069 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672073 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672076 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672080 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672083 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672086 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672089 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672092 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672094 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672097 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672100 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672102 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672105 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672109 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672113 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672116 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672119 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672121 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672124 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672132 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:04.672134 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672135 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672138 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672140 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672143 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672146 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672149 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672151 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672154 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672156 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672159 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672161 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672163 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672166 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672168 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672170 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672173 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672175 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672179 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672182 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672185 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:04.672609 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672187 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672190 2554 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672193 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672195 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672197 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672199 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672202 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672204 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672206 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672209 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672211 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672214 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672217 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672219 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672222 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672225 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672227 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672230 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672232 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:04.673180 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672235 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672238 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672241 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672243 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672246 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672248 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672251 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672254 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672256 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672259 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672262 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672265 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672267 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672270 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672272 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672276 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672278 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672281 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672283 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:04.673643 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672286 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672288 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672291 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672293 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672296 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672298 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672301 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672681 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672686 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672689 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672692 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672694 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672697 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672699 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672702 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672705 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672707 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672709 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672712 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672715 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:04.674113 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672718 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672721 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672723 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672726 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672728 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672731 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672734 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672736 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672738 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672741 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672743 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672745 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672748 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672750 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672753 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672755 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672758 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672761 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672763 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672766 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:04.674605 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672769 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672771 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672774 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672776 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672779 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672781 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672784 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672787 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672790 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672792 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672795 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672798 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672801 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672803 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672806 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672808 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672810 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672813 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672816 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:04.675128 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672820 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672823 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672827 2554 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672846 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672851 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672855 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672858 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672863 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672867 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672871 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672876 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672880 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672882 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672885 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672887 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672890 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672892 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672895 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672897 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672900 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:04.675592 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672903 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672905 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672907 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672910 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672912 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672915 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672918 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672921 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672923 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672926 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672929 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672932 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672934 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.672937 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673022 2554 flags.go:64] FLAG: --address="0.0.0.0" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673036 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673046 2554 flags.go:64] FLAG: --anonymous-auth="true" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673051 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673055 2554 flags.go:64] FLAG: --authentication-token-webhook="false" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673058 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673063 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 20 13:50:04.676132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673068 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673071 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673074 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673078 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673081 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673084 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673087 2554 flags.go:64] FLAG: --cgroup-root="" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673090 2554 flags.go:64] FLAG: --cgroups-per-qos="true" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673093 2554 flags.go:64] FLAG: --client-ca-file="" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673096 2554 flags.go:64] FLAG: --cloud-config="" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673098 2554 flags.go:64] FLAG: --cloud-provider="external" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673101 2554 flags.go:64] FLAG: --cluster-dns="[]" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673105 2554 flags.go:64] FLAG: --cluster-domain="" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673108 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673110 2554 flags.go:64] FLAG: --config-dir="" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673113 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673116 2554 flags.go:64] FLAG: --container-log-max-files="5" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673120 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673123 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673127 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673130 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673133 2554 flags.go:64] FLAG: --contention-profiling="false" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673136 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673138 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673142 2554 flags.go:64] FLAG: --cpu-manager-policy="none" May 20 13:50:04.676723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673144 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673148 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673151 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673154 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673157 2554 flags.go:64] FLAG: --enable-load-reader="false" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673160 2554 flags.go:64] FLAG: --enable-server="true" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673163 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673168 2554 flags.go:64] FLAG: --event-burst="100" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673171 2554 flags.go:64] FLAG: --event-qps="50" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673174 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673177 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673180 2554 flags.go:64] FLAG: --eviction-hard="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673184 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673186 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673190 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673193 2554 flags.go:64] FLAG: --eviction-soft="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673196 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673199 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673202 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673205 2554 flags.go:64] FLAG: --experimental-mounter-path="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673208 2554 flags.go:64] FLAG: --fail-cgroupv1="false" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673211 2554 flags.go:64] FLAG: --fail-swap-on="true" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673214 2554 flags.go:64] FLAG: --feature-gates="" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673217 2554 flags.go:64] FLAG: --file-check-frequency="20s" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673220 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 20 13:50:04.677371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673223 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673227 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673230 2554 flags.go:64] FLAG: --healthz-port="10248" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673233 2554 flags.go:64] FLAG: --help="false" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673236 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-140-251.ec2.internal" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673239 2554 flags.go:64] FLAG: --housekeeping-interval="10s" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673242 2554 flags.go:64] FLAG: --http-check-frequency="20s" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673245 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673248 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673251 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673254 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673257 2554 flags.go:64] FLAG: --image-service-endpoint="" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673260 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673263 2554 flags.go:64] FLAG: --kube-api-burst="100" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673266 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673269 2554 flags.go:64] FLAG: --kube-api-qps="50" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673272 2554 flags.go:64] FLAG: --kube-reserved="" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673275 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673278 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673281 2554 flags.go:64] FLAG: --kubelet-cgroups="" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673284 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673287 2554 flags.go:64] FLAG: --lock-file="" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673289 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673292 2554 flags.go:64] FLAG: --log-flush-frequency="5s" May 20 13:50:04.678865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673295 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673300 2554 flags.go:64] FLAG: --log-json-split-stream="false" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673303 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673306 2554 flags.go:64] FLAG: --log-text-split-stream="false" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673309 2554 flags.go:64] FLAG: --logging-format="text" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673312 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673315 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673318 2554 flags.go:64] FLAG: --manifest-url="" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673321 2554 flags.go:64] FLAG: --manifest-url-header="" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673325 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673330 2554 flags.go:64] FLAG: --max-open-files="1000000" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673334 2554 flags.go:64] FLAG: --max-pods="110" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673337 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673340 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673342 2554 flags.go:64] FLAG: --memory-manager-policy="None" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673345 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673348 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673351 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673354 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673363 2554 flags.go:64] FLAG: --node-status-max-images="50" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673366 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673369 2554 flags.go:64] FLAG: --oom-score-adj="-999" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673372 2554 flags.go:64] FLAG: --pod-cidr="" May 20 13:50:04.679453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673374 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673380 2554 flags.go:64] FLAG: --pod-manifest-path="" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673383 2554 flags.go:64] FLAG: --pod-max-pids="-1" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673386 2554 flags.go:64] FLAG: --pods-per-core="0" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673389 2554 flags.go:64] FLAG: --port="10250" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673391 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673394 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddf9c225577aaaee" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673397 2554 flags.go:64] FLAG: --qos-reserved="" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673400 2554 flags.go:64] FLAG: --read-only-port="10255" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673403 2554 flags.go:64] FLAG: --register-node="true" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673406 2554 flags.go:64] FLAG: --register-schedulable="true" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673409 2554 flags.go:64] FLAG: --register-with-taints="" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673413 2554 flags.go:64] FLAG: --registry-burst="10" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673415 2554 flags.go:64] FLAG: --registry-qps="5" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673418 2554 flags.go:64] FLAG: --reserved-cpus="" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673421 2554 flags.go:64] FLAG: --reserved-memory="" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673425 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673427 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673430 2554 flags.go:64] FLAG: --rotate-certificates="false" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673433 2554 flags.go:64] FLAG: --rotate-server-certificates="false" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673437 2554 flags.go:64] FLAG: --runonce="false" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673440 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673443 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673445 2554 flags.go:64] FLAG: --seccomp-default="false" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673448 2554 flags.go:64] FLAG: --serialize-image-pulls="true" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673451 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 20 13:50:04.680054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673454 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673457 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673461 2554 flags.go:64] FLAG: --storage-driver-password="root" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673463 2554 flags.go:64] FLAG: --storage-driver-secure="false" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673466 2554 flags.go:64] FLAG: --storage-driver-table="stats" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673469 2554 flags.go:64] FLAG: --storage-driver-user="root" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673472 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673475 2554 flags.go:64] FLAG: --sync-frequency="1m0s" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673478 2554 flags.go:64] FLAG: --system-cgroups="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673480 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673485 2554 flags.go:64] FLAG: --system-reserved-cgroup="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673488 2554 flags.go:64] FLAG: --tls-cert-file="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673491 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673495 2554 flags.go:64] FLAG: --tls-min-version="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673497 2554 flags.go:64] FLAG: --tls-private-key-file="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673500 2554 flags.go:64] FLAG: --topology-manager-policy="none" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673503 2554 flags.go:64] FLAG: --topology-manager-policy-options="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673506 2554 flags.go:64] FLAG: --topology-manager-scope="container" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673509 2554 flags.go:64] FLAG: --v="2" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673513 2554 flags.go:64] FLAG: --version="false" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673517 2554 flags.go:64] FLAG: --vmodule="" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673522 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.673525 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673616 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:04.680669 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673620 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673623 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673626 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673630 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673633 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673635 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673638 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673641 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673644 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673647 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673650 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673652 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673655 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673674 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673677 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673680 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673683 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673685 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673688 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673691 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:04.681296 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673693 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673696 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673698 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673701 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673703 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673706 2554 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673709 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673711 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673714 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673716 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673719 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673722 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673729 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673731 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673735 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673737 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673740 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673743 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673745 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673748 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:04.681857 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673750 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673753 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673756 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673758 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673761 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673763 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673766 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673768 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673771 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673774 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673776 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673778 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673781 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673783 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673786 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673788 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673791 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673794 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673798 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:04.682353 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673801 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673804 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673806 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673809 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673812 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673816 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673820 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673823 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673826 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673844 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673847 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673850 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673853 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673855 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673858 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673860 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673863 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673865 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673867 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:04.682812 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673870 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673872 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673875 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673877 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673879 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673882 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.673884 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.674572 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.681754 2554 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.681770 2554 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681820 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681825 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681846 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681850 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681854 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681857 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:04.683306 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681860 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681863 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681865 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681868 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681870 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681873 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681876 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681879 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681881 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681884 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681888 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681892 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681895 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681898 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681900 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681903 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681905 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681908 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681910 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681913 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:04.683713 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681915 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681918 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681920 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681923 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681925 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681938 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681941 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681944 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681946 2554 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681949 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681951 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681954 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681956 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681959 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681961 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681964 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681966 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681969 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681971 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681974 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:04.684227 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681976 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681978 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681981 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681983 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681985 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681987 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681990 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681992 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681995 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681997 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.681999 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682002 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682004 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682007 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682010 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682013 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682015 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682018 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682027 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682029 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:04.684711 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682032 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682035 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682037 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682040 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682042 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682045 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682047 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682050 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682052 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682056 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682061 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682064 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682066 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682069 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682072 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682074 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682077 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682080 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682082 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:04.685243 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682085 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.682090 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682222 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682227 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682230 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682233 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682235 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682238 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682241 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682244 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682247 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682250 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682262 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682264 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682267 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682270 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 20 13:50:04.685781 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682272 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682275 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682277 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682280 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682282 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682285 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682287 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682290 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682292 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682294 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682297 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682299 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682302 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682304 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682306 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682309 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682311 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682314 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682316 2554 feature_gate.go:328] unrecognized feature gate: Example May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682318 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 20 13:50:04.686239 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682321 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682323 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682326 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682329 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682331 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682334 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682337 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682339 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682342 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682349 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682352 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682355 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682357 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682360 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682362 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682365 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682368 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682370 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682373 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 20 13:50:04.686875 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682375 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682378 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682380 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682382 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682385 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682387 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682389 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682392 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682394 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682397 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682399 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682401 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682405 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682409 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682412 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682414 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682417 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682420 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682422 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 20 13:50:04.687394 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682425 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682427 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682429 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682432 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682441 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682445 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682449 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682452 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682455 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682457 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682459 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682462 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682464 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:04.682467 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.682471 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 20 13:50:04.687907 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.683248 2554 server.go:962] "Client rotation is on, will bootstrap in background" May 20 13:50:04.688316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.685395 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 20 13:50:04.688316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.686400 2554 server.go:1019] "Starting client certificate rotation" May 20 13:50:04.688316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.686512 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 20 13:50:04.688316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.686573 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 20 13:50:04.717490 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.717455 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 20 13:50:04.722713 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.722688 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 20 13:50:04.738018 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.737990 2554 log.go:25] "Validated CRI v1 runtime API" May 20 13:50:04.743740 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.743722 2554 log.go:25] "Validated CRI v1 image API" May 20 13:50:04.746591 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.746571 2554 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 20 13:50:04.747384 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.747368 2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 20 13:50:04.751102 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.751075 2554 fs.go:135] Filesystem UUIDs: map[2eab3143-9e26-4548-a752-171e2cbcb146:/dev/nvme0n1p3 476c8aa2-4739-4246-ac1e-73c34f5d4457:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] May 20 13:50:04.751169 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.751101 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 20 13:50:04.755793 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.755678 2554 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:04.755157932 +0000 UTC m=+0.461589985 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101329 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216515b79a82f98c48370f8eb60d75 SystemUUID:ec216515-b79a-82f9-8c48-370f8eb60d75 BootID:b0bd2205-e5d5-43bf-a0be-7e7d52cc1ac8 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:50:2a:79:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:50:2a:79:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:eb:10:58:a6:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 20 13:50:04.756520 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.756510 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 20 13:50:04.756616 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.756604 2554 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 20 13:50:04.759323 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.759270 2554 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 20 13:50:04.759461 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.759303 2554 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 20 13:50:04.759508 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.759475 2554 topology_manager.go:138] "Creating topology manager with none policy" May 20 13:50:04.759508 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.759482 2554 container_manager_linux.go:306] "Creating device plugin manager" May 20 13:50:04.759508 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.759490 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 20 13:50:04.760301 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.760291 2554 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 20 13:50:04.761943 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.761933 2554 state_mem.go:36] "Initialized new in-memory state store" May 20 13:50:04.762059 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.762050 2554 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 20 13:50:04.764891 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.764879 2554 kubelet.go:491] "Attempting to sync node with API server" May 20 13:50:04.764936 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.764901 2554 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 20 13:50:04.764936 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.764913 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 20 13:50:04.764990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.764950 2554 kubelet.go:397] "Adding apiserver pod source" May 20 13:50:04.764990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.764962 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 20 13:50:04.766305 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.766292 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 20 13:50:04.766347 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.766313 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 20 13:50:04.769553 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.769524 2554 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 20 13:50:04.772175 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.772148 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wh8st" May 20 13:50:04.774537 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.774518 2554 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 20 13:50:04.776269 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776254 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 20 13:50:04.776269 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776273 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776281 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776290 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776304 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776313 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776319 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776324 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776333 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776338 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776346 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 20 13:50:04.776371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.776352 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 20 13:50:04.777295 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.777280 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 20 13:50:04.777295 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.777292 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 20 13:50:04.779122 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.779101 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wh8st" May 20 13:50:04.779875 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.779827 2554 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 20 13:50:04.779948 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.779871 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 20 13:50:04.779948 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.779892 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 20 13:50:04.781250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.781236 2554 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 20 13:50:04.781304 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.781274 2554 server.go:1295] "Started kubelet" May 20 13:50:04.781400 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.781360 2554 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 20 13:50:04.781449 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.781392 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 20 13:50:04.781449 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.781429 2554 server_v1.go:47] "podresources" method="list" useActivePods=true May 20 13:50:04.782059 ip-10-0-140-251 systemd[1]: Started Kubernetes Kubelet. May 20 13:50:04.784002 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.783989 2554 server.go:317] "Adding debug handlers to kubelet server" May 20 13:50:04.785143 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.785126 2554 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 20 13:50:04.790030 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.790007 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 20 13:50:04.790684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.790666 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 20 13:50:04.791284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.791124 2554 volume_manager.go:295] "The desired_state_of_world populator starts" May 20 13:50:04.791284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.791138 2554 volume_manager.go:297] "Starting Kubelet Volume Manager" May 20 13:50:04.791284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.791243 2554 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 20 13:50:04.791442 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.791303 2554 reconstruct.go:97] "Volume reconstruction finished" May 20 13:50:04.791442 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.791310 2554 reconciler.go:26] "Reconciler: start to sync state" May 20 13:50:04.792752 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.792732 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:04.792906 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.792815 2554 factory.go:55] Registering systemd factory May 20 13:50:04.793057 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.792946 2554 factory.go:223] Registration of the systemd container factory successfully May 20 13:50:04.793355 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793293 2554 factory.go:153] Registering CRI-O factory May 20 13:50:04.793355 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793307 2554 factory.go:223] Registration of the crio container factory successfully May 20 13:50:04.793565 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793362 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 20 13:50:04.793565 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793383 2554 factory.go:103] Registering Raw factory May 20 13:50:04.793565 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793398 2554 manager.go:1196] Started watching for new ooms in manager May 20 13:50:04.793811 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.793798 2554 manager.go:319] Starting recovery of all containers May 20 13:50:04.794146 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.794128 2554 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:04.797416 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.797396 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-251.ec2.internal\" not found" node="ip-10-0-140-251.ec2.internal" May 20 13:50:04.805873 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.805733 2554 manager.go:324] Recovery completed May 20 13:50:04.810114 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.810101 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:04.811994 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.811980 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:04.812042 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.812009 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:04.812042 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.812019 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:04.812495 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.812482 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" May 20 13:50:04.812550 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.812496 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 20 13:50:04.812550 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.812514 2554 state_mem.go:36] "Initialized new in-memory state store" May 20 13:50:04.814886 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.814874 2554 policy_none.go:49] "None policy: Start" May 20 13:50:04.814928 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.814892 2554 memory_manager.go:186] "Starting memorymanager" policy="None" May 20 13:50:04.814928 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.814902 2554 state_mem.go:35] "Initializing new in-memory state store" May 20 13:50:04.849684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.849618 2554 manager.go:341] "Starting Device Plugin manager" May 20 13:50:04.849684 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.849667 2554 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 20 13:50:04.849684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.849682 2554 server.go:85] "Starting device plugin registration server" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.849974 2554 eviction_manager.go:189] "Eviction manager: starting control loop" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.849986 2554 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.850082 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.850215 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.850229 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.850663 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 20 13:50:04.868865 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.850705 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:04.920418 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.920362 2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 20 13:50:04.921957 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.921939 2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 20 13:50:04.922037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.921969 2554 status_manager.go:230] "Starting to sync pod status with apiserver" May 20 13:50:04.922037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.921991 2554 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 20 13:50:04.922037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.921997 2554 kubelet.go:2451] "Starting kubelet main sync loop" May 20 13:50:04.922164 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.922036 2554 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 20 13:50:04.924535 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.924516 2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:04.950701 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.950675 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:04.951808 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.951790 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:04.951936 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.951827 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:04.951936 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.951860 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:04.951936 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.951893 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-251.ec2.internal" May 20 13:50:04.958783 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:04.958767 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-251.ec2.internal" May 20 13:50:04.958896 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.958790 2554 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-251.ec2.internal\": node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:04.977400 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:04.977372 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.023206 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.023137 2554 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal"] May 20 13:50:05.023286 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.023216 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:05.024645 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.024630 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:05.024728 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.024664 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:05.024728 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.024677 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:05.026267 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.026251 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:05.026461 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.026446 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.026506 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.026475 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:05.027116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027096 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:05.027218 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027097 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:05.027218 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027155 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:05.027218 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027124 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:05.027218 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027176 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:05.027218 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.027198 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:05.028670 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.028656 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:05.028728 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.028680 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 20 13:50:05.029933 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.029910 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientMemory" May 20 13:50:05.030011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.029940 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasNoDiskPressure" May 20 13:50:05.030011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.029955 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeHasSufficientPID" May 20 13:50:05.054872 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.054821 2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-251.ec2.internal\" not found" node="ip-10-0-140-251.ec2.internal" May 20 13:50:05.059693 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.059675 2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-251.ec2.internal\" not found" node="ip-10-0-140-251.ec2.internal" May 20 13:50:05.077747 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.077716 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.093109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.093077 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.093191 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.093114 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.093191 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.093140 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a667969261292a12d7feb41177d18432-config\") pod \"kube-apiserver-proxy-ip-10-0-140-251.ec2.internal\" (UID: \"a667969261292a12d7feb41177d18432\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:05.177993 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.177954 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.193299 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193271 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a667969261292a12d7feb41177d18432-config\") pod \"kube-apiserver-proxy-ip-10-0-140-251.ec2.internal\" (UID: \"a667969261292a12d7feb41177d18432\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:05.193369 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193302 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.193369 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193320 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.193369 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193347 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.193472 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193375 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a667969261292a12d7feb41177d18432-config\") pod \"kube-apiserver-proxy-ip-10-0-140-251.ec2.internal\" (UID: \"a667969261292a12d7feb41177d18432\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:05.193472 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.193386 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/863c25646d9aae7959d45b934f793e2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal\" (UID: \"863c25646d9aae7959d45b934f793e2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.278806 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.278727 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.358280 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.358232 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:05.362995 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.362974 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:05.378997 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.378954 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.479514 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.479465 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.580101 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.580006 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.680540 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.680502 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.686724 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.686694 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 20 13:50:05.686922 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.686907 2554 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 20 13:50:05.686956 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.686919 2554 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 20 13:50:05.780854 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.780808 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.781079 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.781053 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:04 +0000 UTC" deadline="2027-11-25 15:26:36.320808396 +0000 UTC" May 20 13:50:05.781127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.781080 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13297h36m30.539730892s" May 20 13:50:05.782072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.782051 2554 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:05.790151 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.790130 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 20 13:50:05.798721 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.798690 2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 20 13:50:05.847809 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:05.847753 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda667969261292a12d7feb41177d18432.slice/crio-fc56de8d84dacd64ba5daf6a619a03d8b5429556917add8d90fcbd7a8cdfd5fa WatchSource:0}: Error finding container fc56de8d84dacd64ba5daf6a619a03d8b5429556917add8d90fcbd7a8cdfd5fa: Status 404 returned error can't find the container with id fc56de8d84dacd64ba5daf6a619a03d8b5429556917add8d90fcbd7a8cdfd5fa May 20 13:50:05.848067 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:05.848049 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863c25646d9aae7959d45b934f793e2f.slice/crio-85820d2439555d3f6b634474ea559bae9749418b6f28d72947c034e1ebc2fb71 WatchSource:0}: Error finding container 85820d2439555d3f6b634474ea559bae9749418b6f28d72947c034e1ebc2fb71: Status 404 returned error can't find the container with id 85820d2439555d3f6b634474ea559bae9749418b6f28d72947c034e1ebc2fb71 May 20 13:50:05.852232 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.852154 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 13:50:05.881937 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.881898 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:05.908726 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.908693 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7sbb" May 20 13:50:05.917706 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.917682 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7sbb" May 20 13:50:05.924874 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.924804 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" event={"ID":"a667969261292a12d7feb41177d18432","Type":"ContainerStarted","Data":"fc56de8d84dacd64ba5daf6a619a03d8b5429556917add8d90fcbd7a8cdfd5fa"} May 20 13:50:05.925808 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:05.925786 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" event={"ID":"863c25646d9aae7959d45b934f793e2f","Type":"ContainerStarted","Data":"85820d2439555d3f6b634474ea559bae9749418b6f28d72947c034e1ebc2fb71"} May 20 13:50:05.983014 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:05.982975 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:06.083555 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:06.083508 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-251.ec2.internal\" not found" May 20 13:50:06.098507 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.098440 2554 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:06.192248 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.192220 2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" May 20 13:50:06.204376 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.204338 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 20 13:50:06.205504 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.205481 2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" May 20 13:50:06.212205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.212180 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 20 13:50:06.766333 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.766298 2554 apiserver.go:52] "Watching apiserver" May 20 13:50:06.773906 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.773876 2554 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 20 13:50:06.774315 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.774285 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq","openshift-dns/node-resolver-8bd66","openshift-image-registry/node-ca-8pt55","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal","openshift-multus/multus-pccg2","openshift-multus/network-metrics-daemon-54sgt","openshift-network-diagnostics/network-check-target-tgjlf","openshift-network-operator/iptables-alerter-mm4jd","kube-system/konnectivity-agent-pv5cb","kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b82vg","openshift-multus/multus-additional-cni-plugins-m28lx","openshift-ovn-kubernetes/ovnkube-node-95czc"] May 20 13:50:06.776907 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.776877 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.777037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.776930 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.778249 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.778226 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.779493 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.779463 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 20 13:50:06.779600 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.779463 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ww9k6\"" May 20 13:50:06.779776 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.779728 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 20 13:50:06.780054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.780037 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.780145 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.780071 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 20 13:50:06.781470 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781452 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 20 13:50:06.781561 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781516 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 20 13:50:06.781561 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781547 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 20 13:50:06.781797 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781769 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 20 13:50:06.781904 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781890 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 20 13:50:06.781966 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.781905 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 20 13:50:06.782098 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782080 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 20 13:50:06.782159 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782141 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hckg\"" May 20 13:50:06.782209 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782153 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 20 13:50:06.782371 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782355 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lddbp\"" May 20 13:50:06.782482 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782465 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lnq6\"" May 20 13:50:06.782605 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782549 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 20 13:50:06.782605 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.782565 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 20 13:50:06.783204 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.783173 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.783295 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.783262 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.785043 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.785024 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 20 13:50:06.785043 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.785031 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 20 13:50:06.786038 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.786015 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.786400 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.786380 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787247 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787297 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ccb8z\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787598 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ssjdd\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787728 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787852 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.787889 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.788091 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.788296 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.788350 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k9nlc\"" May 20 13:50:06.789284 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.788680 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 20 13:50:06.789796 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.789560 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.791456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.791437 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pccg2" May 20 13:50:06.791573 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.791548 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 20 13:50:06.792745 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.792728 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 20 13:50:06.792876 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.792858 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8xlhp\"" May 20 13:50:06.793026 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.793006 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.793145 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:06.793109 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:06.793569 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.793541 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 20 13:50:06.793569 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.793570 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 20 13:50:06.793850 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.793812 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c94sq\"" May 20 13:50:06.795073 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.795050 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:06.795187 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:06.795109 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:06.802562 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802544 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-system-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.802663 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802574 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-systemd-units\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.802663 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802599 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-slash\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.802663 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802622 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-kubernetes\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802668 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-lib-modules\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3410ff62-7d89-49aa-8a17-149d4444dcc7-serviceca\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-registration-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802736 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802752 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802768 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.802818 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802849 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-etc-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802892 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-log-socket\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802947 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-run\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802975 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74cfe131-dc6a-4e79-a607-527b9b4b07f4-agent-certs\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.802993 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-netns\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803030 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-node-log\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803049 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-netd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803073 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-config\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803093 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-tmp\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803126 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-os-release\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3410ff62-7d89-49aa-8a17-149d4444dcc7-host\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803170 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-multus-certs\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803194 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fj7z\" (UniqueName: \"kubernetes.io/projected/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-kube-api-access-5fj7z\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803210 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-env-overrides\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-cnibin\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ttc\" (UniqueName: \"kubernetes.io/projected/9c269216-da33-46f1-92e5-b1f773b60732-kube-api-access-c5ttc\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803274 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-bin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803295 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-etc-kubernetes\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803316 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-kubelet\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803333 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-var-lib-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803348 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-os-release\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803382 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-k8s-cni-cncf-io\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803411 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-modprobe-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803430 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-socket-dir-parent\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.803456 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803451 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-kubelet\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803473 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64qb\" (UniqueName: \"kubernetes.io/projected/782ab188-fa6f-46cb-bac9-43fe8de7d548-kube-api-access-t64qb\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803513 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-tmp-dir\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803533 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-sys-fs\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803563 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74cfe131-dc6a-4e79-a607-527b9b4b07f4-konnectivity-ca\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803582 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803601 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-cni-binary-copy\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803624 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-systemd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803646 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-system-cni-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803668 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-cnibin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803700 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-multus\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803735 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-bin\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803751 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-systemd\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-socket-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803806 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-netns\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803867 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-daemon-config\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803894 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsks\" (UniqueName: \"kubernetes.io/projected/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-kube-api-access-kvsks\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803918 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgf9f\" (UniqueName: \"kubernetes.io/projected/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-kube-api-access-hgf9f\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803940 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhg7\" (UniqueName: \"kubernetes.io/projected/3410ff62-7d89-49aa-8a17-149d4444dcc7-kube-api-access-7jhg7\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-conf-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.803981 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovn-node-metrics-cert\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804027 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-var-lib-kubelet\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804050 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-host\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804074 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-binary-copy\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804096 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/eafbe133-94fb-459a-911a-0395ad08a491-kube-api-access-slzxv\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-hosts-file\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/476b93ab-c192-445a-b10f-d1a1782ef131-iptables-alerter-script\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804189 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/476b93ab-c192-445a-b10f-d1a1782ef131-host-slash\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-script-lib\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w965\" (UniqueName: \"kubernetes.io/projected/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-kube-api-access-8w965\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804264 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804287 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-hostroot\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.804632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804314 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-ovn\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804410 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-conf\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804439 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-tuned\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804462 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804479 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804495 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysconfig\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804536 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-device-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804567 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh67q\" (UniqueName: \"kubernetes.io/projected/476b93ab-c192-445a-b10f-d1a1782ef131-kube-api-access-mh67q\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-sys\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.805251 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.804625 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-etc-selinux\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.892130 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.892095 2554 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 20 13:50:06.904882 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904845 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ttc\" (UniqueName: \"kubernetes.io/projected/9c269216-da33-46f1-92e5-b1f773b60732-kube-api-access-c5ttc\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904894 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-bin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904913 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-etc-kubernetes\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904928 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-kubelet\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904947 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-var-lib-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.904970 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-os-release\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905002 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-k8s-cni-cncf-io\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905010 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-etc-kubernetes\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905033 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-bin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905046 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905051 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-k8s-cni-cncf-io\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905033 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-kubelet\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905077 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-var-lib-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905086 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-os-release\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905099 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-modprobe-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905147 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-socket-dir-parent\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905170 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-kubelet\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905189 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-modprobe-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905189 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t64qb\" (UniqueName: \"kubernetes.io/projected/782ab188-fa6f-46cb-bac9-43fe8de7d548-kube-api-access-t64qb\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905209 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-socket-dir-parent\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905232 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-tmp-dir\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905243 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-kubelet\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905252 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-sys-fs\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905269 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74cfe131-dc6a-4e79-a607-527b9b4b07f4-konnectivity-ca\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905289 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905314 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-cni-binary-copy\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905326 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-sys-fs\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905340 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-systemd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.905438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905375 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-systemd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905393 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-system-cni-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905396 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905415 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-cnibin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905439 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-multus\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905440 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-system-cni-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905466 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-bin\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905471 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-cnibin\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905503 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-bin\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905497 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-var-lib-cni-multus\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905530 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-systemd\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905567 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-socket-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905582 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-tmp-dir\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905591 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-netns\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905592 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-systemd\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905646 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-netns\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905684 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-daemon-config\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905695 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-socket-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.906183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905717 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsks\" (UniqueName: \"kubernetes.io/projected/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-kube-api-access-kvsks\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905746 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hgf9f\" (UniqueName: \"kubernetes.io/projected/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-kube-api-access-hgf9f\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905771 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhg7\" (UniqueName: \"kubernetes.io/projected/3410ff62-7d89-49aa-8a17-149d4444dcc7-kube-api-access-7jhg7\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905796 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-conf-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905820 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905910 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74cfe131-dc6a-4e79-a607-527b9b4b07f4-konnectivity-ca\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905916 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-cni-binary-copy\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905954 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-conf-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905956 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovn-node-metrics-cert\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.905995 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-var-lib-kubelet\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906014 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906021 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-host\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906070 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-host\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906071 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-binary-copy\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906070 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-var-lib-kubelet\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906104 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/eafbe133-94fb-459a-911a-0395ad08a491-kube-api-access-slzxv\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906143 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-hosts-file\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.907056 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906174 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/476b93ab-c192-445a-b10f-d1a1782ef131-iptables-alerter-script\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906202 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/476b93ab-c192-445a-b10f-d1a1782ef131-host-slash\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906228 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-script-lib\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906228 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/782ab188-fa6f-46cb-bac9-43fe8de7d548-multus-daemon-config\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906248 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906260 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-hosts-file\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906285 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/476b93ab-c192-445a-b10f-d1a1782ef131-host-slash\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906255 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8w965\" (UniqueName: \"kubernetes.io/projected/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-kube-api-access-8w965\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906326 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906356 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-hostroot\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906380 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-ovn\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906404 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-conf\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906427 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-tuned\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906454 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906479 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906504 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysconfig\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906504 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-run-ovn\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.907733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906654 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-binary-copy\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906710 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906748 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906809 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysconfig\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906868 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-hostroot\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906870 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906900 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-device-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906927 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mh67q\" (UniqueName: \"kubernetes.io/projected/476b93ab-c192-445a-b10f-d1a1782ef131-kube-api-access-mh67q\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906941 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-device-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906953 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-sys\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.906977 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-etc-selinux\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907006 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907011 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-d\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907018 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907032 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-system-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907056 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-systemd-units\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907080 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-slash\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.908499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907103 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-kubernetes\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907127 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-lib-modules\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907152 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-sysctl-conf\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907151 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3410ff62-7d89-49aa-8a17-149d4444dcc7-serviceca\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907168 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-sys\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907184 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-registration-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907227 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-systemd-units\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907237 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-slash\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907264 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-registration-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907275 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-etc-selinux\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907281 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907296 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-lib-modules\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907306 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-system-cni-dir\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907311 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907335 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-kubernetes\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907430 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907465 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.909533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907485 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c269216-da33-46f1-92e5-b1f773b60732-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907492 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/476b93ab-c192-445a-b10f-d1a1782ef131-iptables-alerter-script\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907557 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907599 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-etc-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907661 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-log-socket\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907733 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-etc-openvswitch\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:06.907753 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907805 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-run\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:06.907855 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:07.407813227 +0000 UTC m=+3.114245287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907884 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-log-socket\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907852 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3410ff62-7d89-49aa-8a17-149d4444dcc7-serviceca\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907885 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74cfe131-dc6a-4e79-a607-527b9b4b07f4-agent-certs\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907920 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eafbe133-94fb-459a-911a-0395ad08a491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907931 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-netns\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907976 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-node-log\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.907978 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-run-netns\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908004 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-netd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.910357 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908029 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-config\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908037 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-run\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908055 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-tmp\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908092 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-node-log\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908134 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-host-cni-netd\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908174 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-os-release\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908208 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3410ff62-7d89-49aa-8a17-149d4444dcc7-host\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908252 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-os-release\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908299 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-multus-certs\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908330 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5fj7z\" (UniqueName: \"kubernetes.io/projected/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-kube-api-access-5fj7z\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908375 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-env-overrides\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908403 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-cnibin\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908468 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eafbe133-94fb-459a-911a-0395ad08a491-cnibin\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908572 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-script-lib\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908628 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/782ab188-fa6f-46cb-bac9-43fe8de7d548-host-run-multus-certs\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.908662 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3410ff62-7d89-49aa-8a17-149d4444dcc7-host\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.909064 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-env-overrides\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.909179 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovnkube-config\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.911127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.910044 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-etc-tuned\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.912006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.910187 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-ovn-node-metrics-cert\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.912006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.910341 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-tmp\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.913032 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.913005 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ttc\" (UniqueName: \"kubernetes.io/projected/9c269216-da33-46f1-92e5-b1f773b60732-kube-api-access-c5ttc\") pod \"aws-ebs-csi-driver-node-s24fq\" (UID: \"9c269216-da33-46f1-92e5-b1f773b60732\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:06.913559 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.913533 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64qb\" (UniqueName: \"kubernetes.io/projected/782ab188-fa6f-46cb-bac9-43fe8de7d548-kube-api-access-t64qb\") pod \"multus-pccg2\" (UID: \"782ab188-fa6f-46cb-bac9-43fe8de7d548\") " pod="openshift-multus/multus-pccg2" May 20 13:50:06.913951 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.913928 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74cfe131-dc6a-4e79-a607-527b9b4b07f4-agent-certs\") pod \"konnectivity-agent-pv5cb\" (UID: \"74cfe131-dc6a-4e79-a607-527b9b4b07f4\") " pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:06.915015 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.914989 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w965\" (UniqueName: \"kubernetes.io/projected/de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed-kube-api-access-8w965\") pod \"tuned-b82vg\" (UID: \"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed\") " pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:06.915325 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.915298 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhg7\" (UniqueName: \"kubernetes.io/projected/3410ff62-7d89-49aa-8a17-149d4444dcc7-kube-api-access-7jhg7\") pod \"node-ca-8pt55\" (UID: \"3410ff62-7d89-49aa-8a17-149d4444dcc7\") " pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:06.915489 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.915466 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh67q\" (UniqueName: \"kubernetes.io/projected/476b93ab-c192-445a-b10f-d1a1782ef131-kube-api-access-mh67q\") pod \"iptables-alerter-mm4jd\" (UID: \"476b93ab-c192-445a-b10f-d1a1782ef131\") " pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:06.916207 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.916184 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgf9f\" (UniqueName: \"kubernetes.io/projected/10004c10-5b3d-4c5b-a78c-9b54ce64e0a2-kube-api-access-hgf9f\") pod \"node-resolver-8bd66\" (UID: \"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2\") " pod="openshift-dns/node-resolver-8bd66" May 20 13:50:06.916406 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.916387 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsks\" (UniqueName: \"kubernetes.io/projected/90c2ebbf-fd36-46b1-bbe1-1573a66883b4-kube-api-access-kvsks\") pod \"ovnkube-node-95czc\" (UID: \"90c2ebbf-fd36-46b1-bbe1-1573a66883b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:06.917088 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.917052 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/eafbe133-94fb-459a-911a-0395ad08a491-kube-api-access-slzxv\") pod \"multus-additional-cni-plugins-m28lx\" (UID: \"eafbe133-94fb-459a-911a-0395ad08a491\") " pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:06.918468 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.918434 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:05 +0000 UTC" deadline="2027-12-21 09:28:29.830337015 +0000 UTC" May 20 13:50:06.918558 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.918470 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13915h38m22.911870606s" May 20 13:50:06.920971 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.920947 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fj7z\" (UniqueName: \"kubernetes.io/projected/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-kube-api-access-5fj7z\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:06.925518 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:06.925496 2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:07.009396 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.009362 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:07.015427 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.015395 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:07.015427 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.015425 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:07.015427 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.015436 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:07.015730 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.015520 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:07.515497002 +0000 UTC m=+3.221929058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:07.090418 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.090340 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mm4jd" May 20 13:50:07.097353 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.097304 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:07.106320 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.106295 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:07.112110 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.112083 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b82vg" May 20 13:50:07.118697 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.118673 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m28lx" May 20 13:50:07.125313 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.125289 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bd66" May 20 13:50:07.132110 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.132086 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pt55" May 20 13:50:07.138805 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.138777 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" May 20 13:50:07.143504 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.143482 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pccg2" May 20 13:50:07.197555 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.197522 2554 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 20 13:50:07.411248 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.411224 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:07.411384 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.411366 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:07.411457 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.411446 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:08.411430101 +0000 UTC m=+4.117862144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:07.426708 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.426675 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cfe131_dc6a_4e79_a607_527b9b4b07f4.slice/crio-cf06f083023cb7f56c44f7cb2e0cb34a93b5928f593d95ae338afe9fd2b5e059 WatchSource:0}: Error finding container cf06f083023cb7f56c44f7cb2e0cb34a93b5928f593d95ae338afe9fd2b5e059: Status 404 returned error can't find the container with id cf06f083023cb7f56c44f7cb2e0cb34a93b5928f593d95ae338afe9fd2b5e059 May 20 13:50:07.431560 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.431533 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10004c10_5b3d_4c5b_a78c_9b54ce64e0a2.slice/crio-4242d0c5bf3db4102977ef604622c839f2efc66f91b72dfc26d4177c0d9cb12e WatchSource:0}: Error finding container 4242d0c5bf3db4102977ef604622c839f2efc66f91b72dfc26d4177c0d9cb12e: Status 404 returned error can't find the container with id 4242d0c5bf3db4102977ef604622c839f2efc66f91b72dfc26d4177c0d9cb12e May 20 13:50:07.433019 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.432955 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafbe133_94fb_459a_911a_0395ad08a491.slice/crio-2ab31db5f66f8b987c9642e2c9894d7c6f38da3d57fb90d5e3dfef2907c2d847 WatchSource:0}: Error finding container 2ab31db5f66f8b987c9642e2c9894d7c6f38da3d57fb90d5e3dfef2907c2d847: Status 404 returned error can't find the container with id 2ab31db5f66f8b987c9642e2c9894d7c6f38da3d57fb90d5e3dfef2907c2d847 May 20 13:50:07.433688 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.433660 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c2ebbf_fd36_46b1_bbe1_1573a66883b4.slice/crio-f18b496fbe50048c1d6bec9cc4b8a632e6c6939f1fa59553e567f6ee12ef67c8 WatchSource:0}: Error finding container f18b496fbe50048c1d6bec9cc4b8a632e6c6939f1fa59553e567f6ee12ef67c8: Status 404 returned error can't find the container with id f18b496fbe50048c1d6bec9cc4b8a632e6c6939f1fa59553e567f6ee12ef67c8 May 20 13:50:07.434563 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.434432 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c269216_da33_46f1_92e5_b1f773b60732.slice/crio-4d9d8efcbf789bb2886b91035a682f85ae66016d3991c8467bcf42f8941b9adc WatchSource:0}: Error finding container 4d9d8efcbf789bb2886b91035a682f85ae66016d3991c8467bcf42f8941b9adc: Status 404 returned error can't find the container with id 4d9d8efcbf789bb2886b91035a682f85ae66016d3991c8467bcf42f8941b9adc May 20 13:50:07.435580 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.435553 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782ab188_fa6f_46cb_bac9_43fe8de7d548.slice/crio-65453661d487d77379b6c997f718b4c5e001dc9be71210984a0dc00212d4099e WatchSource:0}: Error finding container 65453661d487d77379b6c997f718b4c5e001dc9be71210984a0dc00212d4099e: Status 404 returned error can't find the container with id 65453661d487d77379b6c997f718b4c5e001dc9be71210984a0dc00212d4099e May 20 13:50:07.437060 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.436614 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3410ff62_7d89_49aa_8a17_149d4444dcc7.slice/crio-452db690dc2bda94e4be01c6f6ce866a24576cc20aded019e97959aa1ac9368f WatchSource:0}: Error finding container 452db690dc2bda94e4be01c6f6ce866a24576cc20aded019e97959aa1ac9368f: Status 404 returned error can't find the container with id 452db690dc2bda94e4be01c6f6ce866a24576cc20aded019e97959aa1ac9368f May 20 13:50:07.440172 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.437872 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5083e0_bd7d_40bc_9c80_9b7a2bdb4bed.slice/crio-9e21b1c64be218017bfaf240372832c6f38d67aa3c60fc1f34b47e023860292e WatchSource:0}: Error finding container 9e21b1c64be218017bfaf240372832c6f38d67aa3c60fc1f34b47e023860292e: Status 404 returned error can't find the container with id 9e21b1c64be218017bfaf240372832c6f38d67aa3c60fc1f34b47e023860292e May 20 13:50:07.440172 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:07.439096 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476b93ab_c192_445a_b10f_d1a1782ef131.slice/crio-fef851783547c4a4aa4036596c5bc7260bdfeb5495d90b8273250db190fe350e WatchSource:0}: Error finding container fef851783547c4a4aa4036596c5bc7260bdfeb5495d90b8273250db190fe350e: Status 404 returned error can't find the container with id fef851783547c4a4aa4036596c5bc7260bdfeb5495d90b8273250db190fe350e May 20 13:50:07.612353 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.612321 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:07.612532 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.612450 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:07.612532 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.612464 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:07.612532 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.612473 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:07.612628 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.612546 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:08.612533132 +0000 UTC m=+4.318965172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:07.918716 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.918655 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:05 +0000 UTC" deadline="2028-01-09 06:23:12.296693239 +0000 UTC" May 20 13:50:07.918716 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.918690 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14368h33m4.378006527s" May 20 13:50:07.922672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.922613 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:07.922868 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:07.922741 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:07.946247 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.946141 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" event={"ID":"a667969261292a12d7feb41177d18432","Type":"ContainerStarted","Data":"7e9ecfc0b662ef158126248d256d17a0cb5ed9eba667e4c627032ec1e33130f6"} May 20 13:50:07.967796 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.967754 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b82vg" event={"ID":"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed","Type":"ContainerStarted","Data":"9e21b1c64be218017bfaf240372832c6f38d67aa3c60fc1f34b47e023860292e"} May 20 13:50:07.971190 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.971130 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pt55" event={"ID":"3410ff62-7d89-49aa-8a17-149d4444dcc7","Type":"ContainerStarted","Data":"452db690dc2bda94e4be01c6f6ce866a24576cc20aded019e97959aa1ac9368f"} May 20 13:50:07.980400 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.980361 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pccg2" event={"ID":"782ab188-fa6f-46cb-bac9-43fe8de7d548","Type":"ContainerStarted","Data":"65453661d487d77379b6c997f718b4c5e001dc9be71210984a0dc00212d4099e"} May 20 13:50:07.989734 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:07.989686 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"f18b496fbe50048c1d6bec9cc4b8a632e6c6939f1fa59553e567f6ee12ef67c8"} May 20 13:50:08.016940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.016796 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" event={"ID":"9c269216-da33-46f1-92e5-b1f773b60732","Type":"ContainerStarted","Data":"4d9d8efcbf789bb2886b91035a682f85ae66016d3991c8467bcf42f8941b9adc"} May 20 13:50:08.030773 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.030684 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerStarted","Data":"2ab31db5f66f8b987c9642e2c9894d7c6f38da3d57fb90d5e3dfef2907c2d847"} May 20 13:50:08.037388 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.037268 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pv5cb" event={"ID":"74cfe131-dc6a-4e79-a607-527b9b4b07f4","Type":"ContainerStarted","Data":"cf06f083023cb7f56c44f7cb2e0cb34a93b5928f593d95ae338afe9fd2b5e059"} May 20 13:50:08.042318 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.041470 2554 generic.go:358] "Generic (PLEG): container finished" podID="863c25646d9aae7959d45b934f793e2f" containerID="12f07bf8ff518627e0911676f972d1a4e2b9e33fd2a3061270ee530f444e6c54" exitCode=0 May 20 13:50:08.042318 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.041548 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" event={"ID":"863c25646d9aae7959d45b934f793e2f","Type":"ContainerDied","Data":"12f07bf8ff518627e0911676f972d1a4e2b9e33fd2a3061270ee530f444e6c54"} May 20 13:50:08.048591 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.048495 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mm4jd" event={"ID":"476b93ab-c192-445a-b10f-d1a1782ef131","Type":"ContainerStarted","Data":"fef851783547c4a4aa4036596c5bc7260bdfeb5495d90b8273250db190fe350e"} May 20 13:50:08.051753 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.051672 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bd66" event={"ID":"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2","Type":"ContainerStarted","Data":"4242d0c5bf3db4102977ef604622c839f2efc66f91b72dfc26d4177c0d9cb12e"} May 20 13:50:08.059091 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.058432 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-251.ec2.internal" podStartSLOduration=2.058413792 podStartE2EDuration="2.058413792s" podCreationTimestamp="2026-05-20 13:50:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:07.965400356 +0000 UTC m=+3.671832419" watchObservedRunningTime="2026-05-20 13:50:08.058413792 +0000 UTC m=+3.764845857" May 20 13:50:08.419516 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.419441 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:08.419695 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.419595 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:08.419695 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.419662 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:10.419642133 +0000 UTC m=+6.126074176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:08.621560 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.621470 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:08.621732 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.621648 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:08.621732 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.621667 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:08.621732 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.621679 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:08.621917 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.621735 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:10.621717547 +0000 UTC m=+6.328149606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:08.926204 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:08.925335 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:08.926204 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:08.925483 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:09.069463 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:09.069425 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" event={"ID":"863c25646d9aae7959d45b934f793e2f","Type":"ContainerStarted","Data":"878f4d111f4b0331bdb112a8da89247dc0c1d62832a5c5c221519e4bfe6cc829"} May 20 13:50:09.922790 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:09.922747 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:09.923188 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:09.923156 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:10.439366 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:10.439321 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:10.439805 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.439525 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:10.439805 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.439593 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:14.43957152 +0000 UTC m=+10.146003575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:10.641174 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:10.641138 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:10.641358 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.641333 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:10.641415 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.641362 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:10.641415 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.641376 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:10.641522 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.641441 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:14.641419689 +0000 UTC m=+10.347851744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:10.923136 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:10.923047 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:10.923300 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:10.923212 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:11.922788 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:11.922749 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:11.923206 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:11.922906 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:12.924615 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:12.924584 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:12.925127 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:12.924763 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:13.922757 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:13.922719 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:13.922953 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:13.922875 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:14.474316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:14.474276 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:14.474784 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.474453 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:14.474784 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.474532 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:22.474510306 +0000 UTC m=+18.180942357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:14.676238 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:14.676196 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:14.676420 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.676366 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:14.676420 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.676392 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:14.676420 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.676406 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:14.676532 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.676476 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:22.676457204 +0000 UTC m=+18.382889249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:14.923648 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:14.923569 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:14.923863 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:14.923664 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:15.922892 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:15.922828 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:15.923327 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:15.922963 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:16.922564 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:16.922522 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:16.922733 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:16.922668 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:17.923156 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:17.923071 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:17.923672 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:17.923186 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:18.922645 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:18.922604 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:18.922807 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:18.922768 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:19.922301 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:19.922263 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:19.922700 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:19.922389 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:20.922428 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:20.922392 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:20.922863 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:20.922541 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:21.923117 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:21.923076 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:21.923565 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:21.923211 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:22.534275 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:22.534226 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:22.534471 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.534390 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:22.534471 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.534463 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.534443106 +0000 UTC m=+34.240875163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:22.736066 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:22.736018 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:22.736217 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.736157 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:22.736217 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.736185 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:22.736217 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.736195 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:22.736333 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.736248 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.736232864 +0000 UTC m=+34.442664904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:22.923412 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:22.923327 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:22.923889 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:22.923486 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:23.922997 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:23.922963 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:23.923261 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:23.923085 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:24.925332 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:24.925290 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:24.925767 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:24.925463 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:25.922185 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:25.922153 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:25.922321 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:25.922271 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:26.101544 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.101354 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b82vg" event={"ID":"de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed","Type":"ContainerStarted","Data":"c12d12958e7b09b7a574549a1613a7157d4d44540eb512b773f437befa0241b1"} May 20 13:50:26.102582 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.102559 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pt55" event={"ID":"3410ff62-7d89-49aa-8a17-149d4444dcc7","Type":"ContainerStarted","Data":"313f0c451da3111421091c2d31086af9bf6696ce82a121dcdcb7458fb84a18d7"} May 20 13:50:26.104065 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.104043 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pv5cb" event={"ID":"74cfe131-dc6a-4e79-a607-527b9b4b07f4","Type":"ContainerStarted","Data":"a4970951382b56a03e0e5da4581640939f3881b8a26b781ebef2ff405995e02b"} May 20 13:50:26.125927 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.125856 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-251.ec2.internal" podStartSLOduration=20.125797994 podStartE2EDuration="20.125797994s" podCreationTimestamp="2026-05-20 13:50:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:09.091063779 +0000 UTC m=+4.797495842" watchObservedRunningTime="2026-05-20 13:50:26.125797994 +0000 UTC m=+21.832230057" May 20 13:50:26.126365 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.126288 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b82vg" podStartSLOduration=3.787140024 podStartE2EDuration="22.126273747s" podCreationTimestamp="2026-05-20 13:50:04 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.439264368 +0000 UTC m=+3.145696421" lastFinishedPulling="2026-05-20 13:50:25.778398089 +0000 UTC m=+21.484830144" observedRunningTime="2026-05-20 13:50:26.126213718 +0000 UTC m=+21.832645761" watchObservedRunningTime="2026-05-20 13:50:26.126273747 +0000 UTC m=+21.832705809" May 20 13:50:26.159423 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.159383 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pv5cb" podStartSLOduration=3.890134997 podStartE2EDuration="22.159367625s" podCreationTimestamp="2026-05-20 13:50:04 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.430510304 +0000 UTC m=+3.136942345" lastFinishedPulling="2026-05-20 13:50:25.699742922 +0000 UTC m=+21.406174973" observedRunningTime="2026-05-20 13:50:26.141168138 +0000 UTC m=+21.847600200" watchObservedRunningTime="2026-05-20 13:50:26.159367625 +0000 UTC m=+21.865799678" May 20 13:50:26.159823 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.159796 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8pt55" podStartSLOduration=2.898448546 podStartE2EDuration="21.159790805s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.43839824 +0000 UTC m=+3.144830294" lastFinishedPulling="2026-05-20 13:50:25.699740511 +0000 UTC m=+21.406172553" observedRunningTime="2026-05-20 13:50:26.159504165 +0000 UTC m=+21.865936224" watchObservedRunningTime="2026-05-20 13:50:26.159790805 +0000 UTC m=+21.866222866" May 20 13:50:26.922826 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:26.922608 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:26.923037 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:26.922892 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:27.106857 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.106812 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pccg2" event={"ID":"782ab188-fa6f-46cb-bac9-43fe8de7d548","Type":"ContainerStarted","Data":"db645dc7f9fd5020bccb45a02d655c50c969ee20f95b2ee10002c6d59031d5b9"} May 20 13:50:27.109658 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.109638 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:50:27.110047 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110020 2554 generic.go:358] "Generic (PLEG): container finished" podID="90c2ebbf-fd36-46b1-bbe1-1573a66883b4" containerID="70da411ac7effb8a0711a6c428f4ae61521f25f12a9c46d9bf973eb45fa253a6" exitCode=1 May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110096 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"a44a96bbf6d1291d07b0203c78bdc80a3a84a1aadb093e88a6fa15a095f03937"} May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110122 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"2329d2da41113617400099bb675ad7a5d091f4d0098bc73ff2cb5a1f4d544e0f"} May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110138 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"55ca6295042d520ee606f1ac69ae428e08c7530570b81031e591da514ce62090"} May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110153 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"86bdf457b191cf2b9cb6fbbbb8233ae14e1bd09bfc86049f281a423f53bd0318"} May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110166 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerDied","Data":"70da411ac7effb8a0711a6c428f4ae61521f25f12a9c46d9bf973eb45fa253a6"} May 20 13:50:27.110277 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.110180 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"bd231d4e148888aac80c71a418949d830b55ef2a7657668a1e5924ef95127598"} May 20 13:50:27.111860 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.111782 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" event={"ID":"9c269216-da33-46f1-92e5-b1f773b60732","Type":"ContainerStarted","Data":"af30b812bedc9abb17e361a006e07c2cb09b8ce56547db950d88249d736427e5"} May 20 13:50:27.113522 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.113498 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="ca433dce7e48a4d185c8beabb7d24603f8c5418a1c112f14bf36a9f88a904abf" exitCode=0 May 20 13:50:27.113620 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.113530 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"ca433dce7e48a4d185c8beabb7d24603f8c5418a1c112f14bf36a9f88a904abf"} May 20 13:50:27.115067 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.115044 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bd66" event={"ID":"10004c10-5b3d-4c5b-a78c-9b54ce64e0a2","Type":"ContainerStarted","Data":"dc6d493067133f02bab0d587c8a294f1307a99e7ca183adc98c3e3a591f452de"} May 20 13:50:27.123905 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.123863 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-pccg2" podStartSLOduration=3.744210297 podStartE2EDuration="22.123848513s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.438009086 +0000 UTC m=+3.144441126" lastFinishedPulling="2026-05-20 13:50:25.8176473 +0000 UTC m=+21.524079342" observedRunningTime="2026-05-20 13:50:27.123489896 +0000 UTC m=+22.829921969" watchObservedRunningTime="2026-05-20 13:50:27.123848513 +0000 UTC m=+22.830280572" May 20 13:50:27.124676 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.124653 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 20 13:50:27.862543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.862428 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:27.124672894Z","UUID":"ad6c42c2-5632-4c0e-8d7a-af9c134b9171","Handler":null,"Name":"","Endpoint":""} May 20 13:50:27.864397 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.864373 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 20 13:50:27.864538 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.864406 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 20 13:50:27.923247 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:27.923210 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:27.923478 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:27.923357 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:28.119406 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:28.119323 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" event={"ID":"9c269216-da33-46f1-92e5-b1f773b60732","Type":"ContainerStarted","Data":"a472f2566abbc0f09da269a500e2ca754b19df99e3015cc7c1100566f142065c"} May 20 13:50:28.120871 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:28.120804 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mm4jd" event={"ID":"476b93ab-c192-445a-b10f-d1a1782ef131","Type":"ContainerStarted","Data":"e18a84d9cbec96d1fd8b5c286923d7eb01b63c8f0a1d8bc335e3cd86e56f85d6"} May 20 13:50:28.136559 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:28.136516 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-8bd66" podStartSLOduration=4.755610525 podStartE2EDuration="23.136498346s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.433200071 +0000 UTC m=+3.139632111" lastFinishedPulling="2026-05-20 13:50:25.814087889 +0000 UTC m=+21.520519932" observedRunningTime="2026-05-20 13:50:27.157345483 +0000 UTC m=+22.863777546" watchObservedRunningTime="2026-05-20 13:50:28.136498346 +0000 UTC m=+23.842930409" May 20 13:50:28.136828 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:28.136798 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mm4jd" podStartSLOduration=5.7999493619999996 podStartE2EDuration="24.136792819s" podCreationTimestamp="2026-05-20 13:50:04 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.441513869 +0000 UTC m=+3.147945926" lastFinishedPulling="2026-05-20 13:50:25.77835733 +0000 UTC m=+21.484789383" observedRunningTime="2026-05-20 13:50:28.136643602 +0000 UTC m=+23.843075659" watchObservedRunningTime="2026-05-20 13:50:28.136792819 +0000 UTC m=+23.843224881" May 20 13:50:28.922847 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:28.922640 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:28.923004 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:28.922943 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:29.125751 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.125671 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:50:29.126201 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.126111 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"630cae6328d526e1edf14559e1c9dfde8cc329e0363146ba6df447483f74524a"} May 20 13:50:29.128129 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.128088 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" event={"ID":"9c269216-da33-46f1-92e5-b1f773b60732","Type":"ContainerStarted","Data":"54b48d8e23b6876e86f0d131ad1d3bb85d8af1cf4d8140b448dc538f61232337"} May 20 13:50:29.145566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.145513 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s24fq" podStartSLOduration=3.315109425 podStartE2EDuration="24.145493925s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.436802949 +0000 UTC m=+3.143235007" lastFinishedPulling="2026-05-20 13:50:28.267187463 +0000 UTC m=+23.973619507" observedRunningTime="2026-05-20 13:50:29.145195634 +0000 UTC m=+24.851627710" watchObservedRunningTime="2026-05-20 13:50:29.145493925 +0000 UTC m=+24.851925987" May 20 13:50:29.170280 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.170245 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:29.170963 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.170942 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:29.922349 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:29.922318 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:29.922524 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:29.922428 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:30.130951 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:30.130701 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:30.131568 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:30.131209 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pv5cb" May 20 13:50:30.923005 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:30.922972 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:30.923197 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:30.923132 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:31.923050 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:31.922851 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:31.923614 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:31.923137 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:32.137582 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.137554 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:50:32.137955 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.137922 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"2eed591f6c1ec69aed8019a49fe1465de0e85717a8d44e5611e8cb8cc6e5e41f"} May 20 13:50:32.138174 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.138154 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:32.138308 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.138286 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:32.138428 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.138329 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:32.138496 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.138447 2554 scope.go:117] "RemoveContainer" containerID="70da411ac7effb8a0711a6c428f4ae61521f25f12a9c46d9bf973eb45fa253a6" May 20 13:50:32.140075 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.140033 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="77b1ad27e88888b8ec42106cff9fc1c5ddc588216e93c4a3998c2d4aee2018c8" exitCode=0 May 20 13:50:32.140156 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.140065 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"77b1ad27e88888b8ec42106cff9fc1c5ddc588216e93c4a3998c2d4aee2018c8"} May 20 13:50:32.156021 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.155997 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:32.156122 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.156066 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:50:32.922382 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:32.922348 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:32.922484 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:32.922463 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:33.096632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.096603 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgjlf"] May 20 13:50:33.097024 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.096738 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:33.097024 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:33.096864 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:33.100038 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.100014 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-54sgt"] May 20 13:50:33.150246 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.150083 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:50:33.150642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.150622 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:33.150746 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.150623 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" event={"ID":"90c2ebbf-fd36-46b1-bbe1-1573a66883b4","Type":"ContainerStarted","Data":"9e64684a1752b0c936af0414b7d13ee79c3b21aa4a9d792e84659ecd5363fff8"} May 20 13:50:33.150746 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:33.150728 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:33.186522 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:33.186467 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" podStartSLOduration=10.759888744 podStartE2EDuration="29.186450922s" podCreationTimestamp="2026-05-20 13:50:04 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.435779058 +0000 UTC m=+3.142211104" lastFinishedPulling="2026-05-20 13:50:25.86234124 +0000 UTC m=+21.568773282" observedRunningTime="2026-05-20 13:50:33.185327637 +0000 UTC m=+28.891759699" watchObservedRunningTime="2026-05-20 13:50:33.186450922 +0000 UTC m=+28.892882984" May 20 13:50:34.154642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:34.154600 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="e741beed36fc7b96ec7226f2b56a80c8ff4c903d8047eba0cf98199e6b3a9860" exitCode=0 May 20 13:50:34.155134 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:34.154714 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"e741beed36fc7b96ec7226f2b56a80c8ff4c903d8047eba0cf98199e6b3a9860"} May 20 13:50:34.923542 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:34.923500 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:34.923746 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:34.923596 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:34.923746 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:34.923640 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:34.923746 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:34.923736 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:35.510429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:35.510354 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bd66_10004c10-5b3d-4c5b-a78c-9b54ce64e0a2/dns-node-resolver/0.log" May 20 13:50:36.161640 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:36.161602 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="47543e83077740ac2275fc971bad8180b4d587a8a7c1eccd7d757c0a51360dd7" exitCode=0 May 20 13:50:36.161815 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:36.161651 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"47543e83077740ac2275fc971bad8180b4d587a8a7c1eccd7d757c0a51360dd7"} May 20 13:50:36.295061 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:36.295035 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8pt55_3410ff62-7d89-49aa-8a17-149d4444dcc7/node-ca/0.log" May 20 13:50:36.922896 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:36.922799 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:36.922896 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:36.922860 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:36.923321 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:36.922960 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:36.923321 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:36.923052 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:38.551169 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:38.551131 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:38.551808 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.551279 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:38.551808 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.551353 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs podName:0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.551336126 +0000 UTC m=+66.257768166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs") pod "network-metrics-daemon-54sgt" (UID: "0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951") : object "openshift-multus"/"metrics-daemon-secret" not registered May 20 13:50:38.752883 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:38.752844 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:38.753060 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.753032 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 20 13:50:38.753060 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.753060 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 20 13:50:38.753205 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.753073 2554 projected.go:194] Error preparing data for projected volume kube-api-access-gd2k6 for pod openshift-network-diagnostics/network-check-target-tgjlf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:38.753205 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.753133 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6 podName:cfd2613a-159a-4cc9-a098-cb3d1fdd0555 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.753116328 +0000 UTC m=+66.459548368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd2k6" (UniqueName: "kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6") pod "network-check-target-tgjlf" (UID: "cfd2613a-159a-4cc9-a098-cb3d1fdd0555") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 20 13:50:38.922373 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:38.922267 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:38.922538 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.922419 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:38.922538 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:38.922460 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:38.922538 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:38.922528 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:40.922488 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:40.922455 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:40.922988 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:40.922590 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:40.922988 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:40.922649 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:40.922988 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:40.922737 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:42.177463 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:42.177273 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="642257d60c2eeed7d925293b00ac2f4fd24dfba79e2cfba46e7a595749f323dd" exitCode=0 May 20 13:50:42.177463 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:42.177342 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"642257d60c2eeed7d925293b00ac2f4fd24dfba79e2cfba46e7a595749f323dd"} May 20 13:50:42.922412 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:42.922338 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:42.922412 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:42.922354 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:42.922588 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:42.922439 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:42.922588 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:42.922573 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:43.182476 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:43.182447 2554 generic.go:358] "Generic (PLEG): container finished" podID="eafbe133-94fb-459a-911a-0395ad08a491" containerID="cc7a69cb5b6bef9cdc0d7367e5bba8f8b0cda3b1682db1ee16c1df06ee23d7f2" exitCode=0 May 20 13:50:43.182856 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:43.182492 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerDied","Data":"cc7a69cb5b6bef9cdc0d7367e5bba8f8b0cda3b1682db1ee16c1df06ee23d7f2"} May 20 13:50:44.186885 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:44.186824 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m28lx" event={"ID":"eafbe133-94fb-459a-911a-0395ad08a491","Type":"ContainerStarted","Data":"07df6b3782c45e6114dacb4a5c5d2594a2f63c8295877f34fe44a7da0e1de1ad"} May 20 13:50:44.211541 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:44.211369 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m28lx" podStartSLOduration=4.985883408 podStartE2EDuration="39.211353777s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:50:07.434763084 +0000 UTC m=+3.141195123" lastFinishedPulling="2026-05-20 13:50:41.660233447 +0000 UTC m=+37.366665492" observedRunningTime="2026-05-20 13:50:44.211007126 +0000 UTC m=+39.917439189" watchObservedRunningTime="2026-05-20 13:50:44.211353777 +0000 UTC m=+39.917785839" May 20 13:50:44.922906 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:44.922877 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:44.923102 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:44.922966 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:44.923102 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:44.923050 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:44.923233 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:44.923154 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:46.923064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:46.923022 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:46.923465 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:46.923027 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:46.923465 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:46.923170 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:46.923465 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:46.923235 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:48.922164 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:48.922131 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:48.922614 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:48.922264 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:48.922614 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:48.922343 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:48.922614 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:48.922510 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:50.922977 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:50.922937 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:50.923424 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:50.922937 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:50.923424 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:50.923090 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:50.923424 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:50.923145 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:52.923014 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:52.922972 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:52.923434 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:52.923028 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:52.923434 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:52.923143 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:52.923434 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:52.923252 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:54.922964 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:54.922932 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:54.923481 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:54.923039 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54sgt" podUID="0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951" May 20 13:50:54.923481 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:54.923128 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:54.923481 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:50:54.923224 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgjlf" podUID="cfd2613a-159a-4cc9-a098-cb3d1fdd0555" May 20 13:50:56.611581 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.611551 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-251.ec2.internal" event="NodeReady" May 20 13:50:56.611967 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.611663 2554 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 20 13:50:56.663468 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.663433 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8qlkz"] May 20 13:50:56.706480 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.706437 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w5knr"] May 20 13:50:56.706652 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.706597 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.709207 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.709180 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 20 13:50:56.709341 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.709209 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qrdp6\"" May 20 13:50:56.709341 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.709208 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 20 13:50:56.719930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.719909 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qlkz"] May 20 13:50:56.719930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.719933 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w5knr"] May 20 13:50:56.720069 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.720031 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.723141 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.723115 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 20 13:50:56.723328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.723151 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 20 13:50:56.723328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.723161 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 20 13:50:56.723328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.723202 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x54bb\"" May 20 13:50:56.723473 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.723463 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 20 13:50:56.771483 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.771404 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xps6t"] May 20 13:50:56.789039 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.789011 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xps6t"] May 20 13:50:56.789179 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.789119 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.793283 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.793259 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 20 13:50:56.793420 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.793328 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lgp7k\"" May 20 13:50:56.793420 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.793331 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 20 13:50:56.793507 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.793259 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 20 13:50:56.877974 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.877932 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/557d1a9e-01a7-4c3c-a75e-4414d543aaba-config-volume\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.878163 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.877989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d1a9e-01a7-4c3c-a75e-4414d543aaba-tmp-dir\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.878163 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878013 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.878163 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878087 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/557d1a9e-01a7-4c3c-a75e-4414d543aaba-metrics-tls\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.878163 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878117 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c96636d-79c8-41f2-bdef-3567fa77eb84-cert\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.878163 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878146 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-data-volume\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.878340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzqw\" (UniqueName: \"kubernetes.io/projected/557d1a9e-01a7-4c3c-a75e-4414d543aaba-kube-api-access-7lzqw\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.878340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878218 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8m9z\" (UniqueName: \"kubernetes.io/projected/2c96636d-79c8-41f2-bdef-3567fa77eb84-kube-api-access-g8m9z\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.878340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878232 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-crio-socket\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.878340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878248 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsxs\" (UniqueName: \"kubernetes.io/projected/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-api-access-mcsxs\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.878474 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.878329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.922902 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.922866 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:50:56.922990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.922874 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:50:56.928961 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.928935 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 20 13:50:56.929465 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.929444 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rh8zp\"" May 20 13:50:56.930109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.930078 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6nbf\"" May 20 13:50:56.930256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.930225 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 20 13:50:56.930256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.930247 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 20 13:50:56.978850 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978790 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/557d1a9e-01a7-4c3c-a75e-4414d543aaba-config-volume\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978872 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d1a9e-01a7-4c3c-a75e-4414d543aaba-tmp-dir\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978894 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978926 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/557d1a9e-01a7-4c3c-a75e-4414d543aaba-metrics-tls\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978953 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c96636d-79c8-41f2-bdef-3567fa77eb84-cert\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.978977 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-data-volume\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979002 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzqw\" (UniqueName: \"kubernetes.io/projected/557d1a9e-01a7-4c3c-a75e-4414d543aaba-kube-api-access-7lzqw\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979260 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979132 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g8m9z\" (UniqueName: \"kubernetes.io/projected/2c96636d-79c8-41f2-bdef-3567fa77eb84-kube-api-access-g8m9z\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.979347 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979165 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-crio-socket\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979407 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979385 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d1a9e-01a7-4c3c-a75e-4414d543aaba-tmp-dir\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979458 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979389 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsxs\" (UniqueName: \"kubernetes.io/projected/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-api-access-mcsxs\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979508 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979465 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/557d1a9e-01a7-4c3c-a75e-4414d543aaba-config-volume\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.979508 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979472 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979604 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979525 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-crio-socket\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979604 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979559 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.979785 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.979768 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-data-volume\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.983096 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.983071 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/557d1a9e-01a7-4c3c-a75e-4414d543aaba-metrics-tls\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:56.983236 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.983079 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:56.983236 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.983162 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c96636d-79c8-41f2-bdef-3567fa77eb84-cert\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.993127 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.993099 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8m9z\" (UniqueName: \"kubernetes.io/projected/2c96636d-79c8-41f2-bdef-3567fa77eb84-kube-api-access-g8m9z\") pod \"ingress-canary-xps6t\" (UID: \"2c96636d-79c8-41f2-bdef-3567fa77eb84\") " pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:56.996217 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:56.996178 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzqw\" (UniqueName: \"kubernetes.io/projected/557d1a9e-01a7-4c3c-a75e-4414d543aaba-kube-api-access-7lzqw\") pod \"dns-default-8qlkz\" (UID: \"557d1a9e-01a7-4c3c-a75e-4414d543aaba\") " pod="openshift-dns/dns-default-8qlkz" May 20 13:50:57.009945 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.009915 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsxs\" (UniqueName: \"kubernetes.io/projected/ee6476bf-ec84-412b-ab6f-bc8ab4040ecf-kube-api-access-mcsxs\") pod \"insights-runtime-extractor-w5knr\" (UID: \"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf\") " pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:57.015848 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.015810 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qlkz" May 20 13:50:57.028567 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.028541 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w5knr" May 20 13:50:57.098596 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.098565 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xps6t" May 20 13:50:57.196852 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.196801 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w5knr"] May 20 13:50:57.205112 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.204690 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qlkz"] May 20 13:50:57.210894 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:57.210850 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557d1a9e_01a7_4c3c_a75e_4414d543aaba.slice/crio-e70de4de9b8cc4993dad20649170764ae1752c4aae2611e79824b32f9e3ff354 WatchSource:0}: Error finding container e70de4de9b8cc4993dad20649170764ae1752c4aae2611e79824b32f9e3ff354: Status 404 returned error can't find the container with id e70de4de9b8cc4993dad20649170764ae1752c4aae2611e79824b32f9e3ff354 May 20 13:50:57.213948 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.213920 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5knr" event={"ID":"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf","Type":"ContainerStarted","Data":"b3f0e55af8b802052bfb5439960408f344079e7d085af81deef43ba22eca5c57"} May 20 13:50:57.214883 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.214858 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qlkz" event={"ID":"557d1a9e-01a7-4c3c-a75e-4414d543aaba","Type":"ContainerStarted","Data":"e70de4de9b8cc4993dad20649170764ae1752c4aae2611e79824b32f9e3ff354"} May 20 13:50:57.235532 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:57.235506 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xps6t"] May 20 13:50:57.248762 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:50:57.248736 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c96636d_79c8_41f2_bdef_3567fa77eb84.slice/crio-5b4a125ace95c86a95b6c18b4ff2936330e8a0e65e15510c7f0ff90387e7ecec WatchSource:0}: Error finding container 5b4a125ace95c86a95b6c18b4ff2936330e8a0e65e15510c7f0ff90387e7ecec: Status 404 returned error can't find the container with id 5b4a125ace95c86a95b6c18b4ff2936330e8a0e65e15510c7f0ff90387e7ecec May 20 13:50:58.220463 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:58.220414 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5knr" event={"ID":"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf","Type":"ContainerStarted","Data":"4c7493d93cd9650b0a58c92dc0133bc6243469c6533e8d3e6ee7ec73a784d1ff"} May 20 13:50:58.222876 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:50:58.222827 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xps6t" event={"ID":"2c96636d-79c8-41f2-bdef-3567fa77eb84","Type":"ContainerStarted","Data":"5b4a125ace95c86a95b6c18b4ff2936330e8a0e65e15510c7f0ff90387e7ecec"} May 20 13:51:00.229440 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.229355 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5knr" event={"ID":"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf","Type":"ContainerStarted","Data":"57f85ad9f5de5d34b2394df80727432d02bf160ca4b65fa87ceb0ed70d6f0bfa"} May 20 13:51:00.230827 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.230802 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qlkz" event={"ID":"557d1a9e-01a7-4c3c-a75e-4414d543aaba","Type":"ContainerStarted","Data":"355900141a618a6ac9e918a04791bab6b43a37ccd3452e4263a39c7b023673e9"} May 20 13:51:00.230962 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.230856 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qlkz" event={"ID":"557d1a9e-01a7-4c3c-a75e-4414d543aaba","Type":"ContainerStarted","Data":"b2f38185015f5bfd5dce761502db3fcb54b95350dd1ab7aed8b43db287e009e6"} May 20 13:51:00.231009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.230963 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8qlkz" May 20 13:51:00.232044 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.232024 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xps6t" event={"ID":"2c96636d-79c8-41f2-bdef-3567fa77eb84","Type":"ContainerStarted","Data":"4d711994932f941e179de0ce514ebcaa1f1592f9fee53ce128997db71cc2c2a5"} May 20 13:51:00.250989 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:00.250947 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-8qlkz" podStartSLOduration=1.728417028 podStartE2EDuration="4.250934553s" podCreationTimestamp="2026-05-20 13:50:56 +0000 UTC" firstStartedPulling="2026-05-20 13:50:57.212671747 +0000 UTC m=+52.919103786" lastFinishedPulling="2026-05-20 13:50:59.735189257 +0000 UTC m=+55.441621311" observedRunningTime="2026-05-20 13:51:00.250490171 +0000 UTC m=+55.956922233" watchObservedRunningTime="2026-05-20 13:51:00.250934553 +0000 UTC m=+55.957366621" May 20 13:51:02.239666 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:02.239623 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5knr" event={"ID":"ee6476bf-ec84-412b-ab6f-bc8ab4040ecf","Type":"ContainerStarted","Data":"2c6418ba3c0e6bfa2e61d0bfc167e7216046771231375a46927d8fae450e2dba"} May 20 13:51:02.258126 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:02.258070 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xps6t" podStartSLOduration=3.768121566 podStartE2EDuration="6.258053461s" podCreationTimestamp="2026-05-20 13:50:56 +0000 UTC" firstStartedPulling="2026-05-20 13:50:57.250642373 +0000 UTC m=+52.957074413" lastFinishedPulling="2026-05-20 13:50:59.740574263 +0000 UTC m=+55.447006308" observedRunningTime="2026-05-20 13:51:00.269959916 +0000 UTC m=+55.976391980" watchObservedRunningTime="2026-05-20 13:51:02.258053461 +0000 UTC m=+57.964485505" May 20 13:51:02.258661 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:02.258634 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w5knr" podStartSLOduration=2.068377961 podStartE2EDuration="6.258627678s" podCreationTimestamp="2026-05-20 13:50:56 +0000 UTC" firstStartedPulling="2026-05-20 13:50:57.330947848 +0000 UTC m=+53.037379889" lastFinishedPulling="2026-05-20 13:51:01.521197561 +0000 UTC m=+57.227629606" observedRunningTime="2026-05-20 13:51:02.25739627 +0000 UTC m=+57.963828345" watchObservedRunningTime="2026-05-20 13:51:02.258627678 +0000 UTC m=+57.965059719" May 20 13:51:03.281470 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.281434 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s"] May 20 13:51:03.300673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.300642 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s"] May 20 13:51:03.300858 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.300751 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:03.303140 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.303115 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" May 20 13:51:03.303337 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.303314 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2dr4g\"" May 20 13:51:03.431153 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.431108 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c6d82d7-d37c-42bc-a3f0-6793334bfd0c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-96t9s\" (UID: \"3c6d82d7-d37c-42bc-a3f0-6793334bfd0c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:03.532150 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.532062 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c6d82d7-d37c-42bc-a3f0-6793334bfd0c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-96t9s\" (UID: \"3c6d82d7-d37c-42bc-a3f0-6793334bfd0c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:03.534572 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.534541 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c6d82d7-d37c-42bc-a3f0-6793334bfd0c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-96t9s\" (UID: \"3c6d82d7-d37c-42bc-a3f0-6793334bfd0c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:03.609658 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.609623 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:03.747090 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:03.747055 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s"] May 20 13:51:03.750672 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:03.750643 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6d82d7_d37c_42bc_a3f0_6793334bfd0c.slice/crio-5aa36181706699559c0e4e48476d5d53251765c46c11e3f2314d73bd309b9eab WatchSource:0}: Error finding container 5aa36181706699559c0e4e48476d5d53251765c46c11e3f2314d73bd309b9eab: Status 404 returned error can't find the container with id 5aa36181706699559c0e4e48476d5d53251765c46c11e3f2314d73bd309b9eab May 20 13:51:04.166393 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:04.166364 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95czc" May 20 13:51:04.245958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:04.245921 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" event={"ID":"3c6d82d7-d37c-42bc-a3f0-6793334bfd0c","Type":"ContainerStarted","Data":"5aa36181706699559c0e4e48476d5d53251765c46c11e3f2314d73bd309b9eab"} May 20 13:51:05.255685 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:05.255591 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" event={"ID":"3c6d82d7-d37c-42bc-a3f0-6793334bfd0c","Type":"ContainerStarted","Data":"0b5cfd80efac71646a91244f5cb64c84ebfe32608230d7fc7a652411c6b97036"} May 20 13:51:05.256125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:05.255869 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:05.260515 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:05.260493 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" May 20 13:51:05.271061 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:05.271019 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-96t9s" podStartSLOduration=1.086074938 podStartE2EDuration="2.27100506s" podCreationTimestamp="2026-05-20 13:51:03 +0000 UTC" firstStartedPulling="2026-05-20 13:51:03.752755016 +0000 UTC m=+59.459187056" lastFinishedPulling="2026-05-20 13:51:04.937685123 +0000 UTC m=+60.644117178" observedRunningTime="2026-05-20 13:51:05.270466127 +0000 UTC m=+60.976898189" watchObservedRunningTime="2026-05-20 13:51:05.27100506 +0000 UTC m=+60.977437125" May 20 13:51:06.331854 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.331803 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-68fxv"] May 20 13:51:06.333791 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.333776 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.336106 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336078 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 20 13:51:06.336106 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336090 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" May 20 13:51:06.336106 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336099 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6fstv\"" May 20 13:51:06.336620 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336111 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" May 20 13:51:06.336620 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336111 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 20 13:51:06.336620 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.336540 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 20 13:51:06.342175 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.342155 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-68fxv"] May 20 13:51:06.352275 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.352254 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54685d75-0eac-4f7a-8abc-ebbdda07ff40-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.352408 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.352298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.352408 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.352329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.352408 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.352398 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptvc\" (UniqueName: \"kubernetes.io/projected/54685d75-0eac-4f7a-8abc-ebbdda07ff40-kube-api-access-6ptvc\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.452968 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.452932 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.453109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.452982 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.453109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.453040 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptvc\" (UniqueName: \"kubernetes.io/projected/54685d75-0eac-4f7a-8abc-ebbdda07ff40-kube-api-access-6ptvc\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.453109 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.453101 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54685d75-0eac-4f7a-8abc-ebbdda07ff40-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.453873 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.453765 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54685d75-0eac-4f7a-8abc-ebbdda07ff40-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.455588 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.455557 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.455588 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.455560 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54685d75-0eac-4f7a-8abc-ebbdda07ff40-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.461632 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.461612 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptvc\" (UniqueName: \"kubernetes.io/projected/54685d75-0eac-4f7a-8abc-ebbdda07ff40-kube-api-access-6ptvc\") pod \"prometheus-operator-7779bbdd78-68fxv\" (UID: \"54685d75-0eac-4f7a-8abc-ebbdda07ff40\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.643280 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.643168 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" May 20 13:51:06.771516 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:06.771483 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-68fxv"] May 20 13:51:06.774418 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:06.774391 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54685d75_0eac_4f7a_8abc_ebbdda07ff40.slice/crio-be6e58c854f8024d3411f6e1e97182990c18073899b21566e8b4f39ea0454cc8 WatchSource:0}: Error finding container be6e58c854f8024d3411f6e1e97182990c18073899b21566e8b4f39ea0454cc8: Status 404 returned error can't find the container with id be6e58c854f8024d3411f6e1e97182990c18073899b21566e8b4f39ea0454cc8 May 20 13:51:07.262408 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:07.262362 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" event={"ID":"54685d75-0eac-4f7a-8abc-ebbdda07ff40","Type":"ContainerStarted","Data":"be6e58c854f8024d3411f6e1e97182990c18073899b21566e8b4f39ea0454cc8"} May 20 13:51:08.266817 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:08.266737 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" event={"ID":"54685d75-0eac-4f7a-8abc-ebbdda07ff40","Type":"ContainerStarted","Data":"72edb6df5bf36f6c15e602e3649d7891b58cb5d80ba3650f19eb510300092e17"} May 20 13:51:08.266817 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:08.266772 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" event={"ID":"54685d75-0eac-4f7a-8abc-ebbdda07ff40","Type":"ContainerStarted","Data":"d1341774733537997e358b59bd47a95cc92bf833ac1b826cd6692da814bc22b8"} May 20 13:51:08.285870 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:08.285787 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-68fxv" podStartSLOduration=1.059691331 podStartE2EDuration="2.285771588s" podCreationTimestamp="2026-05-20 13:51:06 +0000 UTC" firstStartedPulling="2026-05-20 13:51:06.776358386 +0000 UTC m=+62.482790426" lastFinishedPulling="2026-05-20 13:51:08.002438644 +0000 UTC m=+63.708870683" observedRunningTime="2026-05-20 13:51:08.285526172 +0000 UTC m=+63.991958236" watchObservedRunningTime="2026-05-20 13:51:08.285771588 +0000 UTC m=+63.992203654" May 20 13:51:09.662879 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.662826 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l"] May 20 13:51:09.665024 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.664998 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.666302 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.666275 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2"] May 20 13:51:09.667482 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.667456 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" May 20 13:51:09.667604 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.667526 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" May 20 13:51:09.667604 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.667530 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lq5s7\"" May 20 13:51:09.668070 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.668053 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.670385 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.670369 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" May 20 13:51:09.671137 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.671107 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" May 20 13:51:09.671248 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.671145 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c584k\"" May 20 13:51:09.671248 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.671193 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" May 20 13:51:09.674729 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg77\" (UniqueName: \"kubernetes.io/projected/b216f484-8614-4e6d-aa56-0fca48baf258-kube-api-access-5gg77\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.674855 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674765 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.674855 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674793 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rhz\" (UniqueName: \"kubernetes.io/projected/e556af9d-264a-476c-8625-1075d2deb558-kube-api-access-n7rhz\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.674971 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.674971 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674951 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.675055 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.674989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.675055 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.675020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.675125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.675076 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b216f484-8614-4e6d-aa56-0fca48baf258-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.675162 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.675127 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.675197 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.675172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e556af9d-264a-476c-8625-1075d2deb558-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.679598 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.679573 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l"] May 20 13:51:09.683393 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.683345 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2"] May 20 13:51:09.692212 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.692193 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tp5tq"] May 20 13:51:09.694191 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.694177 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.696501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.696478 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 20 13:51:09.696501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.696481 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 20 13:51:09.696918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.696901 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wkpl\"" May 20 13:51:09.697124 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.697106 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 20 13:51:09.776254 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776215 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.776429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.776429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776312 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.776429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776348 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.776429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776376 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.776601 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.776469 2554 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found May 20 13:51:09.776601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776467 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-textfile\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.776601 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.776549 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls podName:e556af9d-264a-476c-8625-1075d2deb558 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.276528475 +0000 UTC m=+65.982960514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-6gt5l" (UID: "e556af9d-264a-476c-8625-1075d2deb558") : secret "openshift-state-metrics-tls" not found May 20 13:51:09.776601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776591 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-metrics-client-ca\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.776783 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776629 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b216f484-8614-4e6d-aa56-0fca48baf258-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.776783 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776653 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-sys\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.776783 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.776783 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776717 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg77\" (UniqueName: \"kubernetes.io/projected/b216f484-8614-4e6d-aa56-0fca48baf258-kube-api-access-5gg77\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.776974 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776894 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rhz\" (UniqueName: \"kubernetes.io/projected/e556af9d-264a-476c-8625-1075d2deb558-kube-api-access-n7rhz\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.776974 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776951 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.777092 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.776982 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-wtmp\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.777173 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777077 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b216f484-8614-4e6d-aa56-0fca48baf258-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.777210 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777089 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.777210 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777182 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.777267 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-root\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.777267 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777248 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptrt\" (UniqueName: \"kubernetes.io/projected/c9798965-fae2-4cee-ba67-986c20612bdd-kube-api-access-mptrt\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.777322 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777275 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.777322 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777294 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e556af9d-264a-476c-8625-1075d2deb558-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.777380 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777330 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.777414 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.777398 2554 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found May 20 13:51:09.777467 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.777456 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls podName:b216f484-8614-4e6d-aa56-0fca48baf258 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.277438667 +0000 UTC m=+65.983870710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-lxmn2" (UID: "b216f484-8614-4e6d-aa56-0fca48baf258") : secret "kube-state-metrics-tls" not found May 20 13:51:09.777784 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.777764 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e556af9d-264a-476c-8625-1075d2deb558-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.779080 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.779061 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.779567 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.779551 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.786770 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.786738 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg77\" (UniqueName: \"kubernetes.io/projected/b216f484-8614-4e6d-aa56-0fca48baf258-kube-api-access-5gg77\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:09.786901 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.786874 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rhz\" (UniqueName: \"kubernetes.io/projected/e556af9d-264a-476c-8625-1075d2deb558-kube-api-access-n7rhz\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:09.877930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.877897 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-sys\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.877940 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.877975 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.877995 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-wtmp\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878001 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-sys\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878019 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-root\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878046 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mptrt\" (UniqueName: \"kubernetes.io/projected/c9798965-fae2-4cee-ba67-986c20612bdd-kube-api-access-mptrt\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878094 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-root\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878128 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878165 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-textfile\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878177 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-wtmp\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878192 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-metrics-client-ca\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.878254 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found May 20 13:51:09.878342 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:09.878322 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls podName:c9798965-fae2-4cee-ba67-986c20612bdd nodeName:}" failed. No retries permitted until 2026-05-20 13:51:10.37830164 +0000 UTC m=+66.084733688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls") pod "node-exporter-tp5tq" (UID: "c9798965-fae2-4cee-ba67-986c20612bdd") : secret "node-exporter-tls" not found May 20 13:51:09.878755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878592 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.878755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878713 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9798965-fae2-4cee-ba67-986c20612bdd-metrics-client-ca\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.879004 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.878983 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-textfile\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.880631 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.880614 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:09.892096 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:09.892077 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptrt\" (UniqueName: \"kubernetes.io/projected/c9798965-fae2-4cee-ba67-986c20612bdd-kube-api-access-mptrt\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:10.236938 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.236901 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8qlkz" May 20 13:51:10.281596 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.281564 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:10.281767 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.281642 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:10.284337 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.284312 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b216f484-8614-4e6d-aa56-0fca48baf258-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-lxmn2\" (UID: \"b216f484-8614-4e6d-aa56-0fca48baf258\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:10.284453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.284312 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e556af9d-264a-476c-8625-1075d2deb558-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6gt5l\" (UID: \"e556af9d-264a-476c-8625-1075d2deb558\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:10.382643 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.382586 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:10.385030 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.385003 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9798965-fae2-4cee-ba67-986c20612bdd-node-exporter-tls\") pod \"node-exporter-tp5tq\" (UID: \"c9798965-fae2-4cee-ba67-986c20612bdd\") " pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:10.576208 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.576130 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" May 20 13:51:10.580953 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.580912 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" May 20 13:51:10.583763 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.583744 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:51:10.586543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.586520 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 20 13:51:10.596925 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.596903 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951-metrics-certs\") pod \"network-metrics-daemon-54sgt\" (UID: \"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951\") " pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:51:10.602872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.602851 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tp5tq" May 20 13:51:10.611004 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:10.610968 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9798965_fae2_4cee_ba67_986c20612bdd.slice/crio-3ebc85545dcc66614ec8d1c4335297864e780b9649472846e92731b9058f315b WatchSource:0}: Error finding container 3ebc85545dcc66614ec8d1c4335297864e780b9649472846e92731b9058f315b: Status 404 returned error can't find the container with id 3ebc85545dcc66614ec8d1c4335297864e780b9649472846e92731b9058f315b May 20 13:51:10.740089 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.740033 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l"] May 20 13:51:10.743054 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.743025 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6nbf\"" May 20 13:51:10.751031 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.751008 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54sgt" May 20 13:51:10.773984 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.773956 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2"] May 20 13:51:10.776357 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:10.776326 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb216f484_8614_4e6d_aa56_0fca48baf258.slice/crio-ebd5778ecab62514514c02efd0a821c2adcc80a844a28546c0c19e20ebff862a WatchSource:0}: Error finding container ebd5778ecab62514514c02efd0a821c2adcc80a844a28546c0c19e20ebff862a: Status 404 returned error can't find the container with id ebd5778ecab62514514c02efd0a821c2adcc80a844a28546c0c19e20ebff862a May 20 13:51:10.785635 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.785604 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:51:10.785916 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.785896 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:51:10.788131 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.788105 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 20 13:51:10.794601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.794468 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.797958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.797664 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 20 13:51:10.797958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.797705 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrgcz\"" May 20 13:51:10.797958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.797713 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 20 13:51:10.797958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.797669 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798166 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798331 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798392 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798706 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798704 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.798828 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 20 13:51:10.799194 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.799104 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 20 13:51:10.813402 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.811735 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2k6\" (UniqueName: \"kubernetes.io/projected/cfd2613a-159a-4cc9-a098-cb3d1fdd0555-kube-api-access-gd2k6\") pod \"network-check-target-tgjlf\" (UID: \"cfd2613a-159a-4cc9-a098-cb3d1fdd0555\") " pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:51:10.813402 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.812000 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:51:10.886864 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886817 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886894 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886920 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886949 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886972 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.886989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887094 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887151 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887190 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4q6r\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887389 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887260 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887389 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.887454 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.887365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.916388 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.916300 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-54sgt"] May 20 13:51:10.921679 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:10.921649 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8fcfe5_0beb_4501_81d5_f9ba0bfa0951.slice/crio-dd69cabb898b3b7d106b1d3b516a2c9989f0bcfb0e52f349623a3b990f948956 WatchSource:0}: Error finding container dd69cabb898b3b7d106b1d3b516a2c9989f0bcfb0e52f349623a3b990f948956: Status 404 returned error can't find the container with id dd69cabb898b3b7d106b1d3b516a2c9989f0bcfb0e52f349623a3b990f948956 May 20 13:51:10.988139 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988105 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988139 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988145 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988162 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988283 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988435 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988332 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988435 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988354 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988435 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988401 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988443 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988484 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988509 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4q6r\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988532 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988563 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.988863 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.988596 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.990563 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.989364 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.990563 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.990191 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.990563 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:10.990291 2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found May 20 13:51:10.990563 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:10.990365 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls podName:ec69018f-a87c-4b7c-bac9-5be3614ac9be nodeName:}" failed. No retries permitted until 2026-05-20 13:51:11.49034408 +0000 UTC m=+67.196776127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be") : secret "alertmanager-main-tls" not found May 20 13:51:10.990563 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.990480 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.991566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.991428 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.991751 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.991731 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.992035 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.992009 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.992153 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.992082 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.992220 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.992184 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.992534 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.992508 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.993558 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.993515 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.995349 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.995328 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:10.999770 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:10.999751 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4q6r\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:11.037604 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.037578 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rh8zp\"" May 20 13:51:11.045763 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.045737 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:51:11.180902 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.180871 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgjlf"] May 20 13:51:11.183796 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:11.183758 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd2613a_159a_4cc9_a098_cb3d1fdd0555.slice/crio-ee92b67c01669d3839aa1baf70a706cbc2786cf5e062d75ae82ff267cb7defec WatchSource:0}: Error finding container ee92b67c01669d3839aa1baf70a706cbc2786cf5e062d75ae82ff267cb7defec: Status 404 returned error can't find the container with id ee92b67c01669d3839aa1baf70a706cbc2786cf5e062d75ae82ff267cb7defec May 20 13:51:11.281499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.281461 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgjlf" event={"ID":"cfd2613a-159a-4cc9-a098-cb3d1fdd0555","Type":"ContainerStarted","Data":"ee92b67c01669d3839aa1baf70a706cbc2786cf5e062d75ae82ff267cb7defec"} May 20 13:51:11.282732 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.282702 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tp5tq" event={"ID":"c9798965-fae2-4cee-ba67-986c20612bdd","Type":"ContainerStarted","Data":"3ebc85545dcc66614ec8d1c4335297864e780b9649472846e92731b9058f315b"} May 20 13:51:11.283883 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.283856 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54sgt" event={"ID":"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951","Type":"ContainerStarted","Data":"dd69cabb898b3b7d106b1d3b516a2c9989f0bcfb0e52f349623a3b990f948956"} May 20 13:51:11.285077 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.285051 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" event={"ID":"b216f484-8614-4e6d-aa56-0fca48baf258","Type":"ContainerStarted","Data":"ebd5778ecab62514514c02efd0a821c2adcc80a844a28546c0c19e20ebff862a"} May 20 13:51:11.286727 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.286705 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" event={"ID":"e556af9d-264a-476c-8625-1075d2deb558","Type":"ContainerStarted","Data":"d2901e0d654f5b43c9257e9d975028d9c22eeba45481ba019d4f0f3fa6bf77c1"} May 20 13:51:11.286801 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.286736 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" event={"ID":"e556af9d-264a-476c-8625-1075d2deb558","Type":"ContainerStarted","Data":"fe47393fa299253be6a2942a323aca61d85cbd441cfc7bdb6dfdd850e31f6dd1"} May 20 13:51:11.286801 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.286749 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" event={"ID":"e556af9d-264a-476c-8625-1075d2deb558","Type":"ContainerStarted","Data":"a5157254cbde5d18c09097d076721a9e4591d04c185c7ffb9dd39b21555d97e1"} May 20 13:51:11.493398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.493267 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:11.496715 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.496660 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:11.731869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:11.731795 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:51:12.292033 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.291996 2554 generic.go:358] "Generic (PLEG): container finished" podID="c9798965-fae2-4cee-ba67-986c20612bdd" containerID="5ac027e547341cbc9f5e3ce861de9ef6f9c11f79effb9858a53570131453171d" exitCode=0 May 20 13:51:12.292484 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.292047 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tp5tq" event={"ID":"c9798965-fae2-4cee-ba67-986c20612bdd","Type":"ContainerDied","Data":"5ac027e547341cbc9f5e3ce861de9ef6f9c11f79effb9858a53570131453171d"} May 20 13:51:12.790742 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.789569 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59b769ccb6-h2bgz"] May 20 13:51:12.792682 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.792607 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.798861 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.795486 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" May 20 13:51:12.798861 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.796346 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2lbjx\"" May 20 13:51:12.798861 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.796743 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" May 20 13:51:12.798861 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.797171 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" May 20 13:51:12.804453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.800501 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" May 20 13:51:12.804453 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.800758 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-tqtui6g0udk3\"" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805653 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805708 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-grpc-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805742 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805770 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805817 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805877 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6152e737-50b6-46c0-a338-403c90da3446-metrics-client-ca\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805925 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.806214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.805992 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj7n\" (UniqueName: \"kubernetes.io/projected/6152e737-50b6-46c0-a338-403c90da3446-kube-api-access-hzj7n\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.807804 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.807608 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" May 20 13:51:12.813158 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.813076 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b769ccb6-h2bgz"] May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906548 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906612 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj7n\" (UniqueName: \"kubernetes.io/projected/6152e737-50b6-46c0-a338-403c90da3446-kube-api-access-hzj7n\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906645 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906684 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-grpc-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906713 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906741 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906782 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.906810 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6152e737-50b6-46c0-a338-403c90da3446-metrics-client-ca\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.907672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.907381 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6152e737-50b6-46c0-a338-403c90da3446-metrics-client-ca\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.911600 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.910190 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-grpc-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.911600 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.911555 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-tls\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.912261 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.911967 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.913064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.912571 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.913064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.912922 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.913064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.913007 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6152e737-50b6-46c0-a338-403c90da3446-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.937193 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.937151 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj7n\" (UniqueName: \"kubernetes.io/projected/6152e737-50b6-46c0-a338-403c90da3446-kube-api-access-hzj7n\") pod \"thanos-querier-59b769ccb6-h2bgz\" (UID: \"6152e737-50b6-46c0-a338-403c90da3446\") " pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:12.943006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:12.942969 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:51:13.180403 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.180365 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:13.297924 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.297236 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54sgt" event={"ID":"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951","Type":"ContainerStarted","Data":"f585c698b5802b304514f8d237d5064af4e76ad47e1f721187beca9f772f448d"} May 20 13:51:13.297924 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.297278 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54sgt" event={"ID":"0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951","Type":"ContainerStarted","Data":"a8d6e3b02912acf9d683acf217cde0635340c39b7fed83302e41e518702f1d78"} May 20 13:51:13.300114 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.300048 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" event={"ID":"b216f484-8614-4e6d-aa56-0fca48baf258","Type":"ContainerStarted","Data":"ef58d7fda26566d86843269ab02e621ebb216f49cc13d9668f0c9fa58b2d40bf"} May 20 13:51:13.300114 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.300077 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" event={"ID":"b216f484-8614-4e6d-aa56-0fca48baf258","Type":"ContainerStarted","Data":"9b8e6813dbc99fc1898924abd69d01e760a38fd11def38750bde26bc5a6eb405"} May 20 13:51:13.300114 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.300086 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" event={"ID":"b216f484-8614-4e6d-aa56-0fca48baf258","Type":"ContainerStarted","Data":"acc1a5bfb5c32e434bf1af2e97c1db75e15904113804c52df54869358f9fead1"} May 20 13:51:13.302006 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.301963 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"fe3e68d8df74ec35b4923f36912c7e4d293ca59bc501419c2d6e9a88128da616"} May 20 13:51:13.305421 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.305367 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" event={"ID":"e556af9d-264a-476c-8625-1075d2deb558","Type":"ContainerStarted","Data":"9982babb49133c43c24c631f3be96c55f5533d86a90797161775c8d61a864c47"} May 20 13:51:13.309568 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.309540 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tp5tq" event={"ID":"c9798965-fae2-4cee-ba67-986c20612bdd","Type":"ContainerStarted","Data":"f68d28bd272167e05c868c054218b688f640e7b714e2f01c8c7679b51c3397c2"} May 20 13:51:13.309663 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.309575 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tp5tq" event={"ID":"c9798965-fae2-4cee-ba67-986c20612bdd","Type":"ContainerStarted","Data":"67a3018f316100f7d11202aad5d6fc98864385aa4af15604d564d462755dc1a8"} May 20 13:51:13.316751 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.316697 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-54sgt" podStartSLOduration=66.483520182 podStartE2EDuration="1m8.316679988s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:51:10.924073671 +0000 UTC m=+66.630505721" lastFinishedPulling="2026-05-20 13:51:12.757233484 +0000 UTC m=+68.463665527" observedRunningTime="2026-05-20 13:51:13.315624599 +0000 UTC m=+69.022056661" watchObservedRunningTime="2026-05-20 13:51:13.316679988 +0000 UTC m=+69.023112052" May 20 13:51:13.342953 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.340655 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6gt5l" podStartSLOduration=2.4884627999999998 podStartE2EDuration="4.340636609s" podCreationTimestamp="2026-05-20 13:51:09 +0000 UTC" firstStartedPulling="2026-05-20 13:51:10.908641645 +0000 UTC m=+66.615073688" lastFinishedPulling="2026-05-20 13:51:12.760815445 +0000 UTC m=+68.467247497" observedRunningTime="2026-05-20 13:51:13.340309423 +0000 UTC m=+69.046741486" watchObservedRunningTime="2026-05-20 13:51:13.340636609 +0000 UTC m=+69.047068672" May 20 13:51:13.342953 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.342925 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b769ccb6-h2bgz"] May 20 13:51:13.348646 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:13.348612 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6152e737_50b6_46c0_a338_403c90da3446.slice/crio-9c1bbbaee0a6feb70e27cd84b0a46e911bcb77272adb9260dffd06ac31426b92 WatchSource:0}: Error finding container 9c1bbbaee0a6feb70e27cd84b0a46e911bcb77272adb9260dffd06ac31426b92: Status 404 returned error can't find the container with id 9c1bbbaee0a6feb70e27cd84b0a46e911bcb77272adb9260dffd06ac31426b92 May 20 13:51:13.364322 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.364247 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-lxmn2" podStartSLOduration=2.386260685 podStartE2EDuration="4.364232657s" podCreationTimestamp="2026-05-20 13:51:09 +0000 UTC" firstStartedPulling="2026-05-20 13:51:10.77846369 +0000 UTC m=+66.484895732" lastFinishedPulling="2026-05-20 13:51:12.75643566 +0000 UTC m=+68.462867704" observedRunningTime="2026-05-20 13:51:13.363442033 +0000 UTC m=+69.069874095" watchObservedRunningTime="2026-05-20 13:51:13.364232657 +0000 UTC m=+69.070664719" May 20 13:51:13.383975 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:13.383920 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tp5tq" podStartSLOduration=3.30722535 podStartE2EDuration="4.383901622s" podCreationTimestamp="2026-05-20 13:51:09 +0000 UTC" firstStartedPulling="2026-05-20 13:51:10.612744921 +0000 UTC m=+66.319176964" lastFinishedPulling="2026-05-20 13:51:11.689421193 +0000 UTC m=+67.395853236" observedRunningTime="2026-05-20 13:51:13.382655124 +0000 UTC m=+69.089087187" watchObservedRunningTime="2026-05-20 13:51:13.383901622 +0000 UTC m=+69.090333683" May 20 13:51:14.020518 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.020482 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76fdb89bc9-jzzrd"] May 20 13:51:14.022894 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.022867 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.026191 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.026167 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" May 20 13:51:14.026311 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.026216 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" May 20 13:51:14.027148 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.027118 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nqspz\"" May 20 13:51:14.027260 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.027123 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" May 20 13:51:14.027260 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.027123 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" May 20 13:51:14.027474 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.027456 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8n0i10fav8dpj\"" May 20 13:51:14.040772 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.040731 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76fdb89bc9-jzzrd"] May 20 13:51:14.118316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-metrics-server-audit-profiles\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4adf3390-37c1-40a6-bea0-e18d040c2db1-audit-log\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118529 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118354 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-client-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118529 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118488 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-client-certs\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118619 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-tls\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118619 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118566 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.118619 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.118590 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swb7k\" (UniqueName: \"kubernetes.io/projected/4adf3390-37c1-40a6-bea0-e18d040c2db1-kube-api-access-swb7k\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219384 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219350 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-metrics-server-audit-profiles\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219398 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4adf3390-37c1-40a6-bea0-e18d040c2db1-audit-log\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219583 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219535 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-client-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219695 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219589 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-client-certs\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219695 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219649 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-tls\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219804 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219713 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219804 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219739 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-swb7k\" (UniqueName: \"kubernetes.io/projected/4adf3390-37c1-40a6-bea0-e18d040c2db1-kube-api-access-swb7k\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.219930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.219807 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4adf3390-37c1-40a6-bea0-e18d040c2db1-audit-log\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.220569 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.220506 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.220742 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.220722 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4adf3390-37c1-40a6-bea0-e18d040c2db1-metrics-server-audit-profiles\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.222654 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.222632 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-client-ca-bundle\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.222944 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.222917 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-tls\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.223085 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.223068 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4adf3390-37c1-40a6-bea0-e18d040c2db1-secret-metrics-server-client-certs\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.228311 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.228286 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swb7k\" (UniqueName: \"kubernetes.io/projected/4adf3390-37c1-40a6-bea0-e18d040c2db1-kube-api-access-swb7k\") pod \"metrics-server-76fdb89bc9-jzzrd\" (UID: \"4adf3390-37c1-40a6-bea0-e18d040c2db1\") " pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.314582 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.314489 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"9c1bbbaee0a6feb70e27cd84b0a46e911bcb77272adb9260dffd06ac31426b92"} May 20 13:51:14.335877 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.335829 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:14.477551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.477500 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-m65x7"] May 20 13:51:14.479879 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.479827 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:14.482855 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.482588 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-bhhhf\"" May 20 13:51:14.482855 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.482710 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" May 20 13:51:14.489341 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.489319 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-m65x7"] May 20 13:51:14.523106 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.523072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-m65x7\" (UID: \"242eeef9-b130-4cdf-91a5-431230b35128\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:14.624362 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.624279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-m65x7\" (UID: \"242eeef9-b130-4cdf-91a5-431230b35128\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:14.624521 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:14.624446 2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found May 20 13:51:14.624574 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:51:14.624523 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert podName:242eeef9-b130-4cdf-91a5-431230b35128 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:15.124505997 +0000 UTC m=+70.830938037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-m65x7" (UID: "242eeef9-b130-4cdf-91a5-431230b35128") : secret "monitoring-plugin-cert" not found May 20 13:51:14.878945 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.878389 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f99946db-kvw7t"] May 20 13:51:14.882923 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.882241 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.884933 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.884792 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" May 20 13:51:14.884933 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.884792 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" May 20 13:51:14.884933 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.884916 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" May 20 13:51:14.885191 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.885127 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-55nlz\"" May 20 13:51:14.885421 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.885402 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" May 20 13:51:14.886727 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.885876 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" May 20 13:51:14.893246 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.893205 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" May 20 13:51:14.893499 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.893467 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f99946db-kvw7t"] May 20 13:51:14.915272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.915245 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76fdb89bc9-jzzrd"] May 20 13:51:14.928283 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928247 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-serving-certs-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928394 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928304 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-metrics-client-ca\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928394 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928343 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928491 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928403 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928491 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928435 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928491 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhll9\" (UniqueName: \"kubernetes.io/projected/066525f6-48de-4c84-b9c3-357fa78d8c2a-kube-api-access-nhll9\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928625 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928503 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.928625 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:14.928552 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-federate-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:14.950746 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:14.950712 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adf3390_37c1_40a6_bea0_e18d040c2db1.slice/crio-620a4884f1810dc906d56b8e14fa7bb8c90bbb8f010018737cc57978a5dd106d WatchSource:0}: Error finding container 620a4884f1810dc906d56b8e14fa7bb8c90bbb8f010018737cc57978a5dd106d: Status 404 returned error can't find the container with id 620a4884f1810dc906d56b8e14fa7bb8c90bbb8f010018737cc57978a5dd106d May 20 13:51:15.029538 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029503 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-serving-certs-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029547 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-metrics-client-ca\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029578 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029627 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029652 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029680 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nhll9\" (UniqueName: \"kubernetes.io/projected/066525f6-48de-4c84-b9c3-357fa78d8c2a-kube-api-access-nhll9\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.029730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029706 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.030050 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.029748 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-federate-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.030767 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.030333 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-metrics-client-ca\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.030767 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.030333 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-serving-certs-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.030962 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.030899 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.032732 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.032704 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-telemeter-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.032864 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.032784 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.033064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.033042 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-secret-telemeter-client\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.033398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.033382 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/066525f6-48de-4c84-b9c3-357fa78d8c2a-federate-client-tls\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.038714 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.038688 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhll9\" (UniqueName: \"kubernetes.io/projected/066525f6-48de-4c84-b9c3-357fa78d8c2a-kube-api-access-nhll9\") pod \"telemeter-client-6f99946db-kvw7t\" (UID: \"066525f6-48de-4c84-b9c3-357fa78d8c2a\") " pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.130734 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.130607 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-m65x7\" (UID: \"242eeef9-b130-4cdf-91a5-431230b35128\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:15.133694 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.133669 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/242eeef9-b130-4cdf-91a5-431230b35128-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-m65x7\" (UID: \"242eeef9-b130-4cdf-91a5-431230b35128\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:15.201108 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.201069 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" May 20 13:51:15.320455 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.320364 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388" exitCode=0 May 20 13:51:15.320455 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.320404 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388"} May 20 13:51:15.323346 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.323299 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" event={"ID":"4adf3390-37c1-40a6-bea0-e18d040c2db1","Type":"ContainerStarted","Data":"620a4884f1810dc906d56b8e14fa7bb8c90bbb8f010018737cc57978a5dd106d"} May 20 13:51:15.325020 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.324989 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgjlf" event={"ID":"cfd2613a-159a-4cc9-a098-cb3d1fdd0555","Type":"ContainerStarted","Data":"be1cc123a6a9b788df97e4b7a956c1f7ad5b03820121e54caf0219ca99b3118b"} May 20 13:51:15.325305 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.325286 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:51:15.362186 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.362132 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f99946db-kvw7t"] May 20 13:51:15.364353 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.364300 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgjlf" podStartSLOduration=66.785248771 podStartE2EDuration="1m10.364280999s" podCreationTimestamp="2026-05-20 13:50:05 +0000 UTC" firstStartedPulling="2026-05-20 13:51:11.186082478 +0000 UTC m=+66.892514518" lastFinishedPulling="2026-05-20 13:51:14.765114695 +0000 UTC m=+70.471546746" observedRunningTime="2026-05-20 13:51:15.362250272 +0000 UTC m=+71.068682333" watchObservedRunningTime="2026-05-20 13:51:15.364280999 +0000 UTC m=+71.070713061" May 20 13:51:15.366522 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:15.366497 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066525f6_48de_4c84_b9c3_357fa78d8c2a.slice/crio-688b5a2c6d0f261ce460b2b54613e2828c6925bd2eda600e59eb5edcde9543b0 WatchSource:0}: Error finding container 688b5a2c6d0f261ce460b2b54613e2828c6925bd2eda600e59eb5edcde9543b0: Status 404 returned error can't find the container with id 688b5a2c6d0f261ce460b2b54613e2828c6925bd2eda600e59eb5edcde9543b0 May 20 13:51:15.395869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.395779 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:15.540156 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:15.538995 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-m65x7"] May 20 13:51:15.543417 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:15.543378 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242eeef9_b130_4cdf_91a5_431230b35128.slice/crio-b1cc571c12714ed38d34b647025b0b92f6fd9e46f9b3551b8db0810cf0a425c0 WatchSource:0}: Error finding container b1cc571c12714ed38d34b647025b0b92f6fd9e46f9b3551b8db0810cf0a425c0: Status 404 returned error can't find the container with id b1cc571c12714ed38d34b647025b0b92f6fd9e46f9b3551b8db0810cf0a425c0 May 20 13:51:16.013014 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.012927 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:16.022750 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.022716 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.026917 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.025668 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" May 20 13:51:16.027813 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.027503 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9xl6n\"" May 20 13:51:16.027991 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.027977 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" May 20 13:51:16.035503 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.035262 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fa3pvi53tc6d5\"" May 20 13:51:16.035650 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.035541 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" May 20 13:51:16.035876 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.035806 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" May 20 13:51:16.036086 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.036070 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" May 20 13:51:16.036282 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.036264 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" May 20 13:51:16.036465 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.036450 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" May 20 13:51:16.038601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.036557 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:16.038601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.036631 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" May 20 13:51:16.038601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.038310 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" May 20 13:51:16.038601 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.038519 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" May 20 13:51:16.047912 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.046671 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" May 20 13:51:16.047912 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.047728 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050125 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050359 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzr5\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050551 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050590 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050638 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050686 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050730 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050788 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050858 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050899 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050963 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.050994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.051872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.051099 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.052644 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.051156 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.052644 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.051195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153307 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153367 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153449 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153493 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153521 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153586 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153612 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153639 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153715 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzr5\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153786 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153820 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.153866 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153871 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153888 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153905 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153957 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.153991 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.154025 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.154057 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.154505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.154099 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.155020 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.154739 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.155020 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.154876 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.155380 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.155245 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.159125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.158377 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.159125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.158669 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.159125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.158817 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.159531 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.159510 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.160093 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.159949 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.160093 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.160031 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.160607 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.160579 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.161449 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.161405 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.162351 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.162315 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.162559 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.162507 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.163265 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.163242 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.164780 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.164750 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.165469 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.165445 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.169700 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.169663 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzr5\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5\") pod \"prometheus-k8s-0\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.331932 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.331775 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" event={"ID":"242eeef9-b130-4cdf-91a5-431230b35128","Type":"ContainerStarted","Data":"b1cc571c12714ed38d34b647025b0b92f6fd9e46f9b3551b8db0810cf0a425c0"} May 20 13:51:16.333759 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.333731 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" event={"ID":"066525f6-48de-4c84-b9c3-357fa78d8c2a","Type":"ContainerStarted","Data":"688b5a2c6d0f261ce460b2b54613e2828c6925bd2eda600e59eb5edcde9543b0"} May 20 13:51:16.363684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.363642 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:16.964205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:16.964176 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:51:16.970516 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:51:16.970490 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff1e3ac_017e_4b62_8182_5ebc28453333.slice/crio-f1f8442431db3d1ddd6ff2e37ee0d0bbd3f55f2ff709023be3c8547773edf54a WatchSource:0}: Error finding container f1f8442431db3d1ddd6ff2e37ee0d0bbd3f55f2ff709023be3c8547773edf54a: Status 404 returned error can't find the container with id f1f8442431db3d1ddd6ff2e37ee0d0bbd3f55f2ff709023be3c8547773edf54a May 20 13:51:17.343855 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:17.343798 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"345d43d871c4f68c7bcd9d4b11966480d8209152081a890d0cb43d9a51ca7c81"} May 20 13:51:17.344276 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:17.343867 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"38d37fa3598df06d7d5df3947a08fb83792ce414ea62cbcd196406a0441fe395"} May 20 13:51:17.345113 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:17.345083 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"f1f8442431db3d1ddd6ff2e37ee0d0bbd3f55f2ff709023be3c8547773edf54a"} May 20 13:51:17.347387 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:17.346773 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" event={"ID":"4adf3390-37c1-40a6-bea0-e18d040c2db1","Type":"ContainerStarted","Data":"b9c672d3f16fc9095d2afd0e96052a667122f5c8915746abc32f612cd1dd6d8f"} May 20 13:51:18.352104 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.351995 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" event={"ID":"242eeef9-b130-4cdf-91a5-431230b35128","Type":"ContainerStarted","Data":"ab2e8d03ff94d69b4f00091c55be0339ec6620b2f28916a217fb28ff2592da97"} May 20 13:51:18.352529 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.352392 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:18.354458 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.354429 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"ed4c0d496febdb275c74fbb71bdd11ef4f41dfc0adb35fd53210d67d3e531de1"} May 20 13:51:18.356076 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.356050 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" exitCode=0 May 20 13:51:18.356255 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.356234 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} May 20 13:51:18.358813 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.358793 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" May 20 13:51:18.370326 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.370270 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" podStartSLOduration=2.521158823 podStartE2EDuration="4.370255198s" podCreationTimestamp="2026-05-20 13:51:14 +0000 UTC" firstStartedPulling="2026-05-20 13:51:14.953008207 +0000 UTC m=+70.659440252" lastFinishedPulling="2026-05-20 13:51:16.802104576 +0000 UTC m=+72.508536627" observedRunningTime="2026-05-20 13:51:17.366135572 +0000 UTC m=+73.072567635" watchObservedRunningTime="2026-05-20 13:51:18.370255198 +0000 UTC m=+74.076687261" May 20 13:51:18.371226 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:18.371174 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-m65x7" podStartSLOduration=2.40670644 podStartE2EDuration="4.371164922s" podCreationTimestamp="2026-05-20 13:51:14 +0000 UTC" firstStartedPulling="2026-05-20 13:51:15.545932854 +0000 UTC m=+71.252364894" lastFinishedPulling="2026-05-20 13:51:17.510391333 +0000 UTC m=+73.216823376" observedRunningTime="2026-05-20 13:51:18.369151347 +0000 UTC m=+74.075583409" watchObservedRunningTime="2026-05-20 13:51:18.371164922 +0000 UTC m=+74.077596984" May 20 13:51:19.366535 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.366479 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"116d9ebd6b0dbece6d2187f79ff66fa3d0a144bc67448fd6161b27daa98bf8c5"} May 20 13:51:19.367100 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.366543 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"01a5eaf676d5bb5ef7f78c031085229a1931be2d6ef3f82c10490744a09835bd"} May 20 13:51:19.367100 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.366559 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" event={"ID":"6152e737-50b6-46c0-a338-403c90da3446","Type":"ContainerStarted","Data":"c40c28d33c9c2ac29dcb333fd91189abe7204d108474fb31338284919791c2d9"} May 20 13:51:19.367100 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.366663 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:19.369333 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.369285 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" event={"ID":"066525f6-48de-4c84-b9c3-357fa78d8c2a","Type":"ContainerStarted","Data":"3279441b571f531b085e9179abf41e226871d80b2ef9904ef730c8c3652c69a9"} May 20 13:51:19.369457 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.369337 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" event={"ID":"066525f6-48de-4c84-b9c3-357fa78d8c2a","Type":"ContainerStarted","Data":"988eb1f19b3b0f634c66578e5d4dd851cea26aa115e208e815445c8aa90ba198"} May 20 13:51:19.369457 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.369355 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" event={"ID":"066525f6-48de-4c84-b9c3-357fa78d8c2a","Type":"ContainerStarted","Data":"fc1eca43d0efc398be352735f0cda1c3ba52a47c39394bdd44cff2c67c706008"} May 20 13:51:19.372773 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372707 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920"} May 20 13:51:19.372773 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372772 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7"} May 20 13:51:19.372996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372788 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4"} May 20 13:51:19.372996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372802 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32"} May 20 13:51:19.372996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372815 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea"} May 20 13:51:19.372996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.372826 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerStarted","Data":"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e"} May 20 13:51:19.392605 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.392552 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" podStartSLOduration=2.084459814 podStartE2EDuration="7.392534017s" podCreationTimestamp="2026-05-20 13:51:12 +0000 UTC" firstStartedPulling="2026-05-20 13:51:13.351186517 +0000 UTC m=+69.057618563" lastFinishedPulling="2026-05-20 13:51:18.659260713 +0000 UTC m=+74.365692766" observedRunningTime="2026-05-20 13:51:19.390251942 +0000 UTC m=+75.096684004" watchObservedRunningTime="2026-05-20 13:51:19.392534017 +0000 UTC m=+75.098966081" May 20 13:51:19.418197 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.418079 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.755027145 podStartE2EDuration="9.418060085s" podCreationTimestamp="2026-05-20 13:51:10 +0000 UTC" firstStartedPulling="2026-05-20 13:51:12.970274577 +0000 UTC m=+68.676706622" lastFinishedPulling="2026-05-20 13:51:18.633307519 +0000 UTC m=+74.339739562" observedRunningTime="2026-05-20 13:51:19.416925003 +0000 UTC m=+75.123357065" watchObservedRunningTime="2026-05-20 13:51:19.418060085 +0000 UTC m=+75.124492149" May 20 13:51:19.445064 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:19.443712 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f99946db-kvw7t" podStartSLOduration=2.173227842 podStartE2EDuration="5.443690642s" podCreationTimestamp="2026-05-20 13:51:14 +0000 UTC" firstStartedPulling="2026-05-20 13:51:15.368628685 +0000 UTC m=+71.075060725" lastFinishedPulling="2026-05-20 13:51:18.639091474 +0000 UTC m=+74.345523525" observedRunningTime="2026-05-20 13:51:19.43868846 +0000 UTC m=+75.145120518" watchObservedRunningTime="2026-05-20 13:51:19.443690642 +0000 UTC m=+75.150122704" May 20 13:51:21.382991 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:21.382963 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} May 20 13:51:21.382991 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:21.383001 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} May 20 13:51:22.388756 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:22.388716 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} May 20 13:51:22.389170 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:22.388762 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} May 20 13:51:22.389170 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:22.388776 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} May 20 13:51:22.389170 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:22.388789 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerStarted","Data":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} May 20 13:51:22.417207 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:22.417149 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.559518293 podStartE2EDuration="7.417131103s" podCreationTimestamp="2026-05-20 13:51:15 +0000 UTC" firstStartedPulling="2026-05-20 13:51:18.358200666 +0000 UTC m=+74.064632706" lastFinishedPulling="2026-05-20 13:51:21.215813474 +0000 UTC m=+76.922245516" observedRunningTime="2026-05-20 13:51:22.415201773 +0000 UTC m=+78.121633834" watchObservedRunningTime="2026-05-20 13:51:22.417131103 +0000 UTC m=+78.123563166" May 20 13:51:25.383394 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:25.383366 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59b769ccb6-h2bgz" May 20 13:51:26.364521 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:26.364479 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:51:34.336982 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:34.336937 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:34.336982 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:34.336983 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:46.336129 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:46.336096 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgjlf" May 20 13:51:50.387173 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:50.387140 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/init-config-reloader/0.log" May 20 13:51:50.392645 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:50.392617 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/alertmanager/0.log" May 20 13:51:50.492168 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:50.492139 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/config-reloader/0.log" May 20 13:51:50.684116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:50.684084 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/kube-rbac-proxy-web/0.log" May 20 13:51:50.881712 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:50.881683 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/kube-rbac-proxy/0.log" May 20 13:51:51.081719 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:51.081635 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/kube-rbac-proxy-metric/0.log" May 20 13:51:51.280849 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:51.280803 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec69018f-a87c-4b7c-bac9-5be3614ac9be/prom-label-proxy/0.log" May 20 13:51:51.684617 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:51.684583 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-state-metrics/0.log" May 20 13:51:51.883480 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:51.883450 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-rbac-proxy-main/0.log" May 20 13:51:52.081740 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:52.081654 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-rbac-proxy-self/0.log" May 20 13:51:52.281448 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:52.281416 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76fdb89bc9-jzzrd_4adf3390-37c1-40a6-bea0-e18d040c2db1/metrics-server/0.log" May 20 13:51:52.482049 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:52.482018 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-m65x7_242eeef9-b130-4cdf-91a5-431230b35128/monitoring-plugin/0.log" May 20 13:51:53.281743 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:53.281714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/init-textfile/0.log" May 20 13:51:53.483349 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:53.483323 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/node-exporter/0.log" May 20 13:51:53.683602 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:53.683577 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/kube-rbac-proxy/0.log" May 20 13:51:54.341365 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:54.341332 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:54.345250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:54.345226 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76fdb89bc9-jzzrd" May 20 13:51:54.481997 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:54.481962 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/kube-rbac-proxy-main/0.log" May 20 13:51:54.682321 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:54.682294 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/kube-rbac-proxy-self/0.log" May 20 13:51:54.881657 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:54.881627 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/openshift-state-metrics/0.log" May 20 13:51:55.083009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:55.082935 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/init-config-reloader/0.log" May 20 13:51:55.285778 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:55.285750 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/prometheus/0.log" May 20 13:51:55.482287 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:55.482260 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/config-reloader/0.log" May 20 13:51:55.682716 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:55.682686 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/thanos-sidecar/0.log" May 20 13:51:55.881927 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:55.881854 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/kube-rbac-proxy-web/0.log" May 20 13:51:56.084236 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:56.084205 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/kube-rbac-proxy/0.log" May 20 13:51:56.282458 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:56.282428 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ff1e3ac-017e-4b62-8182-5ebc28453333/kube-rbac-proxy-thanos/0.log" May 20 13:51:56.483431 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:56.483394 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-68fxv_54685d75-0eac-4f7a-8abc-ebbdda07ff40/prometheus-operator/0.log" May 20 13:51:56.681756 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:56.681717 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-68fxv_54685d75-0eac-4f7a-8abc-ebbdda07ff40/kube-rbac-proxy/0.log" May 20 13:51:56.883982 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:56.883947 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-96t9s_3c6d82d7-d37c-42bc-a3f0-6793334bfd0c/prometheus-operator-admission-webhook/0.log" May 20 13:51:57.081586 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:57.081509 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/telemeter-client/0.log" May 20 13:51:57.281048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:57.281016 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/reload/0.log" May 20 13:51:57.484286 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:57.484261 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/kube-rbac-proxy/0.log" May 20 13:51:57.681741 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:57.681692 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/thanos-query/0.log" May 20 13:51:57.881293 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:57.881215 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-web/0.log" May 20 13:51:58.081828 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:58.081802 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy/0.log" May 20 13:51:58.280923 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:58.280897 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/prom-label-proxy/0.log" May 20 13:51:58.481516 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:58.481491 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-rules/0.log" May 20 13:51:58.682373 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:51:58.682349 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-metrics/0.log" May 20 13:52:16.364707 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:16.364667 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:16.385276 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:16.385254 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:16.591851 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:16.591816 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:30.076404 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076355 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:30.076959 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076800 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="alertmanager" containerID="cri-o://85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" gracePeriod=120 May 20 13:52:30.076959 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076886 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-metric" containerID="cri-o://944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" gracePeriod=120 May 20 13:52:30.076959 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076898 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-web" containerID="cri-o://75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" gracePeriod=120 May 20 13:52:30.076959 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076921 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="config-reloader" containerID="cri-o://e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" gracePeriod=120 May 20 13:52:30.077187 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.076921 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy" containerID="cri-o://1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" gracePeriod=120 May 20 13:52:30.077187 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.077008 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="prom-label-proxy" containerID="cri-o://e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" gracePeriod=120 May 20 13:52:30.623118 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623078 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" exitCode=0 May 20 13:52:30.623118 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623103 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" exitCode=0 May 20 13:52:30.623118 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623110 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" exitCode=0 May 20 13:52:30.623118 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623115 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" exitCode=0 May 20 13:52:30.623378 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623147 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920"} May 20 13:52:30.623378 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623179 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4"} May 20 13:52:30.623378 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623189 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea"} May 20 13:52:30.623378 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:30.623198 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e"} May 20 13:52:31.312655 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.312633 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.427250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427167 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427203 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427228 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427246 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4q6r\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427291 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427311 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427336 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427358 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427393 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427418 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427455 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427496 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.427551 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427521 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") pod \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\" (UID: \"ec69018f-a87c-4b7c-bac9-5be3614ac9be\") " May 20 13:52:31.428028 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.427906 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:31.428517 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.428406 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:31.428736 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.428708 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:31.430521 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.430494 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out" (OuterVolumeSpecName: "config-out") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:31.431230 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431184 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.431340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431238 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume" (OuterVolumeSpecName: "config-volume") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.431340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431263 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.431461 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431361 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.431461 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431407 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:31.431719 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.431697 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.432380 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.432362 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r" (OuterVolumeSpecName: "kube-api-access-s4q6r") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "kube-api-access-s4q6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:31.435201 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.435174 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.442421 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.442400 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config" (OuterVolumeSpecName: "web-config") pod "ec69018f-a87c-4b7c-bac9-5be3614ac9be" (UID: "ec69018f-a87c-4b7c-bac9-5be3614ac9be"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:31.528521 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528481 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528521 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528515 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528521 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528526 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-main-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528537 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-web-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528546 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-tls-assets\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528556 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-out\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528564 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4q6r\" (UniqueName: \"kubernetes.io/projected/ec69018f-a87c-4b7c-bac9-5be3614ac9be-kube-api-access-s4q6r\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528573 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528582 2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-config-volume\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528590 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec69018f-a87c-4b7c-bac9-5be3614ac9be-metrics-client-ca\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528600 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528609 2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec69018f-a87c-4b7c-bac9-5be3614ac9be-cluster-tls-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.528755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.528617 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec69018f-a87c-4b7c-bac9-5be3614ac9be-alertmanager-main-db\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:31.629330 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629292 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" exitCode=0 May 20 13:52:31.629330 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629329 2554 generic.go:358] "Generic (PLEG): container finished" podID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerID="75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" exitCode=0 May 20 13:52:31.629543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629382 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7"} May 20 13:52:31.629543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629425 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32"} May 20 13:52:31.629543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629437 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec69018f-a87c-4b7c-bac9-5be3614ac9be","Type":"ContainerDied","Data":"fe3e68d8df74ec35b4923f36912c7e4d293ca59bc501419c2d6e9a88128da616"} May 20 13:52:31.629543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629451 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.629543 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.629462 2554 scope.go:117] "RemoveContainer" containerID="e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" May 20 13:52:31.637367 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.637342 2554 scope.go:117] "RemoveContainer" containerID="944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" May 20 13:52:31.644999 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.644981 2554 scope.go:117] "RemoveContainer" containerID="1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" May 20 13:52:31.652448 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.652430 2554 scope.go:117] "RemoveContainer" containerID="75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" May 20 13:52:31.656674 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.656653 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:31.661166 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.661150 2554 scope.go:117] "RemoveContainer" containerID="e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" May 20 13:52:31.662117 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.662076 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:31.667991 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.667975 2554 scope.go:117] "RemoveContainer" containerID="85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" May 20 13:52:31.675331 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.675316 2554 scope.go:117] "RemoveContainer" containerID="5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388" May 20 13:52:31.681814 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.681797 2554 scope.go:117] "RemoveContainer" containerID="e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" May 20 13:52:31.682109 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.682088 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920\": container with ID starting with e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920 not found: ID does not exist" containerID="e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" May 20 13:52:31.682171 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682119 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920"} err="failed to get container status \"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920\": rpc error: code = NotFound desc = could not find container \"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920\": container with ID starting with e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920 not found: ID does not exist" May 20 13:52:31.682171 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682153 2554 scope.go:117] "RemoveContainer" containerID="944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" May 20 13:52:31.682382 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.682368 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7\": container with ID starting with 944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7 not found: ID does not exist" containerID="944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" May 20 13:52:31.682422 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682387 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7"} err="failed to get container status \"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7\": rpc error: code = NotFound desc = could not find container \"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7\": container with ID starting with 944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7 not found: ID does not exist" May 20 13:52:31.682422 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682401 2554 scope.go:117] "RemoveContainer" containerID="1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" May 20 13:52:31.682621 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.682605 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4\": container with ID starting with 1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4 not found: ID does not exist" containerID="1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" May 20 13:52:31.682659 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682626 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4"} err="failed to get container status \"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4\": rpc error: code = NotFound desc = could not find container \"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4\": container with ID starting with 1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4 not found: ID does not exist" May 20 13:52:31.682659 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682639 2554 scope.go:117] "RemoveContainer" containerID="75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" May 20 13:52:31.682914 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.682886 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32\": container with ID starting with 75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32 not found: ID does not exist" containerID="75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" May 20 13:52:31.682983 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682920 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32"} err="failed to get container status \"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32\": rpc error: code = NotFound desc = could not find container \"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32\": container with ID starting with 75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32 not found: ID does not exist" May 20 13:52:31.682983 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.682938 2554 scope.go:117] "RemoveContainer" containerID="e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" May 20 13:52:31.683257 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.683229 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea\": container with ID starting with e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea not found: ID does not exist" containerID="e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" May 20 13:52:31.683332 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683251 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea"} err="failed to get container status \"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea\": rpc error: code = NotFound desc = could not find container \"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea\": container with ID starting with e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea not found: ID does not exist" May 20 13:52:31.683332 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683270 2554 scope.go:117] "RemoveContainer" containerID="85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" May 20 13:52:31.683532 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.683511 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e\": container with ID starting with 85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e not found: ID does not exist" containerID="85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" May 20 13:52:31.683598 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683536 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e"} err="failed to get container status \"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e\": rpc error: code = NotFound desc = could not find container \"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e\": container with ID starting with 85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e not found: ID does not exist" May 20 13:52:31.683598 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683555 2554 scope.go:117] "RemoveContainer" containerID="5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388" May 20 13:52:31.683871 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:31.683821 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388\": container with ID starting with 5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388 not found: ID does not exist" containerID="5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388" May 20 13:52:31.683973 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683874 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388"} err="failed to get container status \"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388\": rpc error: code = NotFound desc = could not find container \"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388\": container with ID starting with 5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388 not found: ID does not exist" May 20 13:52:31.683973 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.683896 2554 scope.go:117] "RemoveContainer" containerID="e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920" May 20 13:52:31.684467 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684445 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920"} err="failed to get container status \"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920\": rpc error: code = NotFound desc = could not find container \"e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920\": container with ID starting with e901761013a10999df6bed7291fedd4cc38d8d295cb1a7f453b0988dd5f34920 not found: ID does not exist" May 20 13:52:31.684467 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684466 2554 scope.go:117] "RemoveContainer" containerID="944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7" May 20 13:52:31.684679 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684662 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7"} err="failed to get container status \"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7\": rpc error: code = NotFound desc = could not find container \"944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7\": container with ID starting with 944db24b94dc6964463a958bceba95edb08397e46fd71f00e323c5165de77bf7 not found: ID does not exist" May 20 13:52:31.684724 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684680 2554 scope.go:117] "RemoveContainer" containerID="1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4" May 20 13:52:31.684916 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684896 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4"} err="failed to get container status \"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4\": rpc error: code = NotFound desc = could not find container \"1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4\": container with ID starting with 1f3528d7433659d5ffcb7c1554f6b3df2bf0b436b76e05bb3948c4e2d7b238a4 not found: ID does not exist" May 20 13:52:31.684969 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.684917 2554 scope.go:117] "RemoveContainer" containerID="75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32" May 20 13:52:31.685136 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685114 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32"} err="failed to get container status \"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32\": rpc error: code = NotFound desc = could not find container \"75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32\": container with ID starting with 75d97cdf45aea5667b2858d420d5a64248408e18dadbc10c7b4c237c82f18c32 not found: ID does not exist" May 20 13:52:31.685136 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685129 2554 scope.go:117] "RemoveContainer" containerID="e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea" May 20 13:52:31.685351 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685334 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea"} err="failed to get container status \"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea\": rpc error: code = NotFound desc = could not find container \"e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea\": container with ID starting with e91252f9406c4b48c2f1c2d169726347c40484ae7422b6f508c200e53cd5ecea not found: ID does not exist" May 20 13:52:31.685407 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685351 2554 scope.go:117] "RemoveContainer" containerID="85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e" May 20 13:52:31.685574 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685553 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e"} err="failed to get container status \"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e\": rpc error: code = NotFound desc = could not find container \"85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e\": container with ID starting with 85c1a263d20370d636ba6a3b0c7e598a83b6e34f36131fa54ee908459d75aa3e not found: ID does not exist" May 20 13:52:31.685644 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685576 2554 scope.go:117] "RemoveContainer" containerID="5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388" May 20 13:52:31.685809 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.685789 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388"} err="failed to get container status \"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388\": rpc error: code = NotFound desc = could not find container \"5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388\": container with ID starting with 5e5b0948db7e787bf1028f7b224243bd62c598144292317bf6475a357a288388 not found: ID does not exist" May 20 13:52:31.697530 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697510 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:31.697853 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697824 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-metric" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697858 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-metric" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697879 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="alertmanager" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697885 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="alertmanager" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697892 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697897 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy" May 20 13:52:31.697903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697904 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="init-config-reloader" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697910 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="init-config-reloader" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697916 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="prom-label-proxy" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697922 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="prom-label-proxy" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697933 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="config-reloader" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697939 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="config-reloader" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697944 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-web" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.697950 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-web" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698008 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698021 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-metric" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698030 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="alertmanager" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698038 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="kube-rbac-proxy-web" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698045 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="config-reloader" May 20 13:52:31.698116 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.698052 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" containerName="prom-label-proxy" May 20 13:52:31.703267 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.703251 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.705671 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705647 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 20 13:52:31.705766 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705691 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 20 13:52:31.705828 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705782 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrgcz\"" May 20 13:52:31.705828 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705808 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 20 13:52:31.705964 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705847 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 20 13:52:31.705964 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.705941 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 20 13:52:31.706059 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.706013 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 20 13:52:31.706059 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.706010 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 20 13:52:31.706506 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.706485 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 20 13:52:31.712975 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.712957 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 20 13:52:31.716261 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.716238 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:31.830849 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830809 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830972 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.830988 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831005 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831048 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831298 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831059 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831298 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831100 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dm9\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-kube-api-access-25dm9\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831298 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831123 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831298 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831165 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.831298 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.831221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932327 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932227 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-25dm9\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-kube-api-access-25dm9\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932327 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932280 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932327 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932316 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932327 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932336 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932363 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932387 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932412 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932452 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932476 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932495 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932520 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932548 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.932673 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932572 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.933157 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.932868 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.933614 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.933589 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.933926 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.933902 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148a0223-2d5e-4b5b-b48d-4220f0801ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.935815 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.935617 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.935815 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.935753 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.935815 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.935753 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.936031 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.935879 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.936031 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.936021 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.936302 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.936278 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.936361 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.936306 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.936397 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.936388 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.937694 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.937673 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148a0223-2d5e-4b5b-b48d-4220f0801ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:31.942155 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:31.942134 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dm9\" (UniqueName: \"kubernetes.io/projected/148a0223-2d5e-4b5b-b48d-4220f0801ab2-kube-api-access-25dm9\") pod \"alertmanager-main-0\" (UID: \"148a0223-2d5e-4b5b-b48d-4220f0801ab2\") " pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:32.012759 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.012724 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 20 13:52:32.160782 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.160748 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 20 13:52:32.164477 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:52:32.164436 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148a0223_2d5e_4b5b_b48d_4220f0801ab2.slice/crio-2134957a12a6b24831eb80811bbffc9de2a00d338f6055a87c4052a23b9abd7e WatchSource:0}: Error finding container 2134957a12a6b24831eb80811bbffc9de2a00d338f6055a87c4052a23b9abd7e: Status 404 returned error can't find the container with id 2134957a12a6b24831eb80811bbffc9de2a00d338f6055a87c4052a23b9abd7e May 20 13:52:32.633737 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.633646 2554 generic.go:358] "Generic (PLEG): container finished" podID="148a0223-2d5e-4b5b-b48d-4220f0801ab2" containerID="c058cd016bf95e7058b1fcf8d5ebd7d8d53a52af528b6ce0b87e27913104a92d" exitCode=0 May 20 13:52:32.634175 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.633738 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerDied","Data":"c058cd016bf95e7058b1fcf8d5ebd7d8d53a52af528b6ce0b87e27913104a92d"} May 20 13:52:32.634175 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.633769 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"2134957a12a6b24831eb80811bbffc9de2a00d338f6055a87c4052a23b9abd7e"} May 20 13:52:32.927797 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:32.927763 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec69018f-a87c-4b7c-bac9-5be3614ac9be" path="/var/lib/kubelet/pods/ec69018f-a87c-4b7c-bac9-5be3614ac9be/volumes" May 20 13:52:33.640546 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640504 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"473f6b689336bd9f83c823085011b5797f4c76f0d0fdcf089f0d2bf07a7f8eca"} May 20 13:52:33.640546 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640539 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"c4acd783560cfcd0c21e80c7b6780bb835ef5a953c775f2a333838f4115b82aa"} May 20 13:52:33.640546 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640548 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"cfd79800d58d338dc8723ea4a884c43fc8cb95cd0e16360bd396b176e608ee85"} May 20 13:52:33.641013 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640557 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"1125c22193ee5b28785118288c27acba304173ea64e3bac6dbc6126819c057dd"} May 20 13:52:33.641013 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640564 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"073d2878c681dce926a3df8ad2976ec05178793a701c1e95f4645f2a12005738"} May 20 13:52:33.641013 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.640571 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148a0223-2d5e-4b5b-b48d-4220f0801ab2","Type":"ContainerStarted","Data":"b6e0c5701161ce603492486407de66113e3bdf909650d30eaa0ad76af09fb6ab"} May 20 13:52:33.666899 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:33.666802 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.666783362 podStartE2EDuration="2.666783362s" podCreationTimestamp="2026-05-20 13:52:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:33.664803102 +0000 UTC m=+149.371235166" watchObservedRunningTime="2026-05-20 13:52:33.666783362 +0000 UTC m=+149.373215424" May 20 13:52:34.314988 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.314945 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:34.315606 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315563 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="thanos-sidecar" containerID="cri-o://76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" gracePeriod=600 May 20 13:52:34.315606 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315598 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="prometheus" containerID="cri-o://03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" gracePeriod=600 May 20 13:52:34.315798 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315541 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy" containerID="cri-o://a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" gracePeriod=600 May 20 13:52:34.315798 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315600 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" gracePeriod=600 May 20 13:52:34.315798 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315581 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-web" containerID="cri-o://3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" gracePeriod=600 May 20 13:52:34.315984 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.315549 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="config-reloader" containerID="cri-o://9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" gracePeriod=600 May 20 13:52:34.573320 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.572707 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:34.649329 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649300 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" exitCode=0 May 20 13:52:34.649329 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649324 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" exitCode=0 May 20 13:52:34.649329 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649331 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" exitCode=0 May 20 13:52:34.649329 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649337 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" exitCode=0 May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649342 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" exitCode=0 May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649335 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649378 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649390 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649400 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649411 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649420 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649347 2554 generic.go:358] "Generic (PLEG): container finished" podID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" exitCode=0 May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649437 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649463 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:34.649869 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.649509 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ff1e3ac-017e-4b62-8182-5ebc28453333","Type":"ContainerDied","Data":"f1f8442431db3d1ddd6ff2e37ee0d0bbd3f55f2ff709023be3c8547773edf54a"} May 20 13:52:34.657607 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657587 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.657768 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657748 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.657940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657780 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.657940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657860 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.657940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657888 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.657940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657907 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.657940 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657933 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657959 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.657990 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658034 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658058 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658083 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658119 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfzr5\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658141 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658157 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658183 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658179 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658777 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658223 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658777 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658251 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.658777 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.658276 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config\") pod \"3ff1e3ac-017e-4b62-8182-5ebc28453333\" (UID: \"3ff1e3ac-017e-4b62-8182-5ebc28453333\") " May 20 13:52:34.659780 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.659417 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:34.660097 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.659966 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:34.661918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.660915 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:34.661918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.661182 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:34.661918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.661616 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:34.661918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.661629 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:34.661918 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.660790 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:52:34.662553 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.662507 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out" (OuterVolumeSpecName: "config-out") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:52:34.662633 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.662570 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.663220 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.663186 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.663564 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.663523 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.663810 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.663783 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config" (OuterVolumeSpecName: "config") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.664443 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.664412 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.664523 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.664450 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.664672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.664642 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5" (OuterVolumeSpecName: "kube-api-access-gfzr5") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "kube-api-access-gfzr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:52:34.664798 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.664686 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.665510 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.665479 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.668072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.668055 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.674444 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.674420 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config" (OuterVolumeSpecName: "web-config") pod "3ff1e3ac-017e-4b62-8182-5ebc28453333" (UID: "3ff1e3ac-017e-4b62-8182-5ebc28453333"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:52:34.675040 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.675024 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.681749 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.681731 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.688320 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.688299 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.695126 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.695105 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.701402 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.701386 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.701636 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.701618 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.701679 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.701645 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.701679 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.701666 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.701928 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.701911 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.701982 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.701933 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.701982 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.701950 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.702177 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.702160 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.702217 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702182 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.702217 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702199 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.702429 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.702411 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.702468 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702434 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.702468 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702448 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.702684 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.702669 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.702730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702686 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.702730 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702699 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.702936 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.702916 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.702987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702940 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.702987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.702952 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.703177 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:52:34.703163 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.703222 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703179 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.703222 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703190 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.703405 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703384 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.703447 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703405 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.703599 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703584 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.703599 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703599 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.703819 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703791 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.703923 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.703820 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.704057 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704040 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.704101 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704058 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.704256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704241 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.704302 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704257 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.704467 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704452 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.704467 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704467 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.704626 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704609 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.704672 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704629 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.704782 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704767 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.704817 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704782 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.704996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704981 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.704996 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.704996 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.705197 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705183 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.705245 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705197 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.705392 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705378 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.705429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705392 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.705610 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705592 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.705667 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705611 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.705851 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705814 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.705851 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.705844 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.706037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706022 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.706037 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706036 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.706233 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706218 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.706272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706234 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.706438 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706420 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.706482 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706440 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.706650 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706629 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.706694 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706650 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.706856 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706824 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.706903 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.706857 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.707040 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707025 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.707077 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707039 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.707250 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707230 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.707300 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707252 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.707434 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707420 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.707434 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707433 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.707594 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707578 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.707634 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707597 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.707773 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707758 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.707812 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707774 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.707963 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707947 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.708011 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.707964 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.708133 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708118 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.708171 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708132 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.708316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708298 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.708316 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708315 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.708505 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708489 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.708547 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708505 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.708687 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708671 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.708723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708686 2554 scope.go:117] "RemoveContainer" containerID="9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989" May 20 13:52:34.708892 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708864 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989"} err="failed to get container status \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": rpc error: code = NotFound desc = could not find container \"9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989\": container with ID starting with 9ba9933f4ff414dc1ffd2b188707f41800d041f001e882389427eb7fd94c2989 not found: ID does not exist" May 20 13:52:34.708892 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.708883 2554 scope.go:117] "RemoveContainer" containerID="a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76" May 20 13:52:34.709117 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709097 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76"} err="failed to get container status \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": rpc error: code = NotFound desc = could not find container \"a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76\": container with ID starting with a159cd5055f57688c8f6c53005e39f148a0b60c35698f872eeaa338f87a95c76 not found: ID does not exist" May 20 13:52:34.709157 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709117 2554 scope.go:117] "RemoveContainer" containerID="3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1" May 20 13:52:34.709324 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709310 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1"} err="failed to get container status \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": rpc error: code = NotFound desc = could not find container \"3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1\": container with ID starting with 3299acebac333a65d7f555e0c2ab518e0d2bf934aafea224f690e805a70c89a1 not found: ID does not exist" May 20 13:52:34.709364 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709324 2554 scope.go:117] "RemoveContainer" containerID="76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc" May 20 13:52:34.709494 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709479 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc"} err="failed to get container status \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": rpc error: code = NotFound desc = could not find container \"76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc\": container with ID starting with 76010473c94a8adc95d0cad2d88e309e2c52206b475575b37c186cbca3592dcc not found: ID does not exist" May 20 13:52:34.709535 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709494 2554 scope.go:117] "RemoveContainer" containerID="9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8" May 20 13:52:34.709659 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709643 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8"} err="failed to get container status \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": rpc error: code = NotFound desc = could not find container \"9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8\": container with ID starting with 9bd876bf050e9397f35738e8877dd135849a36973d80c3b0dd81d4c46f129ce8 not found: ID does not exist" May 20 13:52:34.709696 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709658 2554 scope.go:117] "RemoveContainer" containerID="03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87" May 20 13:52:34.709844 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709813 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87"} err="failed to get container status \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": rpc error: code = NotFound desc = could not find container \"03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87\": container with ID starting with 03222f298bb140c15508167137f3fafe44f309f4acc63325ba5d0b137631ca87 not found: ID does not exist" May 20 13:52:34.709898 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.709848 2554 scope.go:117] "RemoveContainer" containerID="6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d" May 20 13:52:34.710020 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.710006 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d"} err="failed to get container status \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": rpc error: code = NotFound desc = could not find container \"6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d\": container with ID starting with 6c3be6df2b4557fb0045efb8d1c3a35b49d8ae59637e69f9b75050b8802e872d not found: ID does not exist" May 20 13:52:34.760220 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760197 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzr5\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-kube-api-access-gfzr5\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760220 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760222 2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760238 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-config-out\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760253 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760266 2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760278 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760292 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-web-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760303 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff1e3ac-017e-4b62-8182-5ebc28453333-tls-assets\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760314 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760328 2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-grpc-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760341 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-metrics-client-ca\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760355 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760369 2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760383 2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-kube-rbac-proxy\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760396 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760413 2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ff1e3ac-017e-4b62-8182-5ebc28453333-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760427 2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ff1e3ac-017e-4b62-8182-5ebc28453333-secret-metrics-client-certs\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.760860 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.760440 2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ff1e3ac-017e-4b62-8182-5ebc28453333-prometheus-k8s-db\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:52:34.967701 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.967668 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:34.972138 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.972114 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:34.997921 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.997891 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:34.998243 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998208 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998246 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998255 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="prometheus" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998262 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="prometheus" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998271 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="config-reloader" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998280 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="config-reloader" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998293 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="init-config-reloader" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998298 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="init-config-reloader" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998302 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-thanos" May 20 13:52:34.998306 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998308 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-thanos" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998317 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="thanos-sidecar" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998322 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="thanos-sidecar" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998328 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-web" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998333 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-web" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998388 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="config-reloader" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998397 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-web" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998405 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998411 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="thanos-sidecar" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998419 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="prometheus" May 20 13:52:34.998566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:34.998425 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" containerName="kube-rbac-proxy-thanos" May 20 13:52:35.002431 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.002413 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.004894 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.004871 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fa3pvi53tc6d5\"" May 20 13:52:35.005192 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005101 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" May 20 13:52:35.005192 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005114 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9xl6n\"" May 20 13:52:35.005192 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005127 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" May 20 13:52:35.005192 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005153 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" May 20 13:52:35.005328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005165 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" May 20 13:52:35.005328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005208 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" May 20 13:52:35.005328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005124 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" May 20 13:52:35.005328 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005114 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" May 20 13:52:35.005602 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005491 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" May 20 13:52:35.005602 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005506 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" May 20 13:52:35.005602 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.005506 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" May 20 13:52:35.008214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.008194 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" May 20 13:52:35.011566 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.011546 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" May 20 13:52:35.015303 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.015285 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:35.164025 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.163980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config-out\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164025 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164025 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25xk\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-kube-api-access-t25xk\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164055 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164080 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164104 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164125 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164228 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164253 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164279 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-web-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164328 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164349 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164417 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164447 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164467 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164501 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164496 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.164734 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.164519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265424 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265344 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265424 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265384 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265625 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265431 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config-out\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265625 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265462 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t25xk\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-kube-api-access-t25xk\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265625 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265599 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265771 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265648 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265771 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265681 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265771 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265710 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265771 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265742 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265776 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265810 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265874 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-web-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265921 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.265987 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265953 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.265990 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.266057 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.266089 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.266116 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266272 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.266177 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.266516 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.266493 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.267489 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.267465 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.267808 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.267789 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.268327 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.268303 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.268562 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.268517 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config-out\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.269213 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.269192 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.269974 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.269949 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/466e3238-d5f5-4a4d-b2b8-4601e137cb86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.270557 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.270530 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.270658 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.270529 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.270723 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.270672 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.270800 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.270781 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.270858 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.270795 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.271129 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.271107 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.271544 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.271525 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.272047 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.272028 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.272376 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.272356 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/466e3238-d5f5-4a4d-b2b8-4601e137cb86-web-config\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.273549 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.273529 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25xk\" (UniqueName: \"kubernetes.io/projected/466e3238-d5f5-4a4d-b2b8-4601e137cb86-kube-api-access-t25xk\") pod \"prometheus-k8s-0\" (UID: \"466e3238-d5f5-4a4d-b2b8-4601e137cb86\") " pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.312282 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.312258 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:52:35.446541 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.446515 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] May 20 13:52:35.448792 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:52:35.448763 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466e3238_d5f5_4a4d_b2b8_4601e137cb86.slice/crio-a2bb8d07433e330390bad3d9259f0aa94bda8d07c1e20318588683f1e99f74f0 WatchSource:0}: Error finding container a2bb8d07433e330390bad3d9259f0aa94bda8d07c1e20318588683f1e99f74f0: Status 404 returned error can't find the container with id a2bb8d07433e330390bad3d9259f0aa94bda8d07c1e20318588683f1e99f74f0 May 20 13:52:35.654336 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.654304 2554 generic.go:358] "Generic (PLEG): container finished" podID="466e3238-d5f5-4a4d-b2b8-4601e137cb86" containerID="ab24905f837dabb4503f605b85a25ace97c2032a53e527746f197191a3da0ca4" exitCode=0 May 20 13:52:35.654721 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.654384 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerDied","Data":"ab24905f837dabb4503f605b85a25ace97c2032a53e527746f197191a3da0ca4"} May 20 13:52:35.654721 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:35.654412 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"a2bb8d07433e330390bad3d9259f0aa94bda8d07c1e20318588683f1e99f74f0"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.660936 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"c22ec24eaf66f97495855974fc3e354d1eb436b86b672b34da25f3f9d76b378b"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.660976 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"ab7e4018b9c622d4bff4e2242faeff653ab8426587c86a72ca09f7b2707856e3"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.660986 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"673f58c31f56a410e8555a4900adeac4e46d5f06c9ff29439d2c424f16f7665d"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.660994 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"092774605ab687c80445c81b58ee6af4d3685e496612ebd5ef5bd19976f8eadd"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.661001 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"1cc086db1767faa66655a498fb96934de6402f6df35ae4cc33b8425f867a93ed"} May 20 13:52:36.661027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.661009 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"466e3238-d5f5-4a4d-b2b8-4601e137cb86","Type":"ContainerStarted","Data":"ea59534d7b63afa0f4053f839bb8fd3a89557b0a422a0c66066ad38b0a6bae90"} May 20 13:52:36.686790 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.686738 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.686721983 podStartE2EDuration="2.686721983s" podCreationTimestamp="2026-05-20 13:52:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:36.685928505 +0000 UTC m=+152.392360564" watchObservedRunningTime="2026-05-20 13:52:36.686721983 +0000 UTC m=+152.393154026" May 20 13:52:36.927655 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:36.927572 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff1e3ac-017e-4b62-8182-5ebc28453333" path="/var/lib/kubelet/pods/3ff1e3ac-017e-4b62-8182-5ebc28453333/volumes" May 20 13:52:40.313181 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:52:40.313136 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:02.825991 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.825909 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hgn5x"] May 20 13:53:02.828653 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.828632 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:02.831318 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.831296 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 20 13:53:02.837924 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.837899 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgn5x"] May 20 13:53:02.913790 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.913749 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-original-pull-secret\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:02.913790 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.913793 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-kubelet-config\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:02.914034 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:02.913929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-dbus\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.014689 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.014650 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-original-pull-secret\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.014689 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.014693 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-kubelet-config\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.014990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.014739 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-dbus\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.014990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.014851 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-kubelet-config\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.014990 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.014947 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-dbus\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.017190 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.017162 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf1705e-a9b8-4b56-b283-ac216c3b2b2f-original-pull-secret\") pod \"global-pull-secret-syncer-hgn5x\" (UID: \"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f\") " pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.138793 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.138698 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgn5x" May 20 13:53:03.264492 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.264455 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgn5x"] May 20 13:53:03.267702 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:53:03.267667 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf1705e_a9b8_4b56_b283_ac216c3b2b2f.slice/crio-93d569072e27861b7ab58b465d15e75b3d0ced87ab2a14a7b4df9548455bc364 WatchSource:0}: Error finding container 93d569072e27861b7ab58b465d15e75b3d0ced87ab2a14a7b4df9548455bc364: Status 404 returned error can't find the container with id 93d569072e27861b7ab58b465d15e75b3d0ced87ab2a14a7b4df9548455bc364 May 20 13:53:03.748022 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:03.747983 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgn5x" event={"ID":"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f","Type":"ContainerStarted","Data":"93d569072e27861b7ab58b465d15e75b3d0ced87ab2a14a7b4df9548455bc364"} May 20 13:53:07.762380 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:07.762345 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgn5x" event={"ID":"5bf1705e-a9b8-4b56-b283-ac216c3b2b2f","Type":"ContainerStarted","Data":"1e096ad863f8b5e70db4288ab14253dcf70f36b418cfce6bc5469f732d80d3b0"} May 20 13:53:07.779290 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:07.779232 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hgn5x" podStartSLOduration=1.832458599 podStartE2EDuration="5.779215594s" podCreationTimestamp="2026-05-20 13:53:02 +0000 UTC" firstStartedPulling="2026-05-20 13:53:03.269338541 +0000 UTC m=+178.975770581" lastFinishedPulling="2026-05-20 13:53:07.216095534 +0000 UTC m=+182.922527576" observedRunningTime="2026-05-20 13:53:07.777974395 +0000 UTC m=+183.484406458" watchObservedRunningTime="2026-05-20 13:53:07.779215594 +0000 UTC m=+183.485647705" May 20 13:53:35.313473 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:35.313433 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:35.330096 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:35.330070 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:35.863877 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:35.863848 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" May 20 13:53:53.513195 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.513157 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-7djgx"] May 20 13:53:53.515546 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.515521 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.518185 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.518165 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" May 20 13:53:53.518185 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.518173 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" May 20 13:53:53.518495 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.518482 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" May 20 13:53:53.519161 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.519138 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2nvxj\"" May 20 13:53:53.519264 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.519142 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" May 20 13:53:53.528461 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.528439 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-7djgx"] May 20 13:53:53.669942 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.669901 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-certificates\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.670124 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.670046 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwhg\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-kube-api-access-pbwhg\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.771016 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.770917 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwhg\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-kube-api-access-pbwhg\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.771342 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.771324 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-certificates\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.774687 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.774657 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-certificates\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.780305 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.780281 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwhg\" (UniqueName: \"kubernetes.io/projected/16d294da-b1d4-4e2c-a6bd-d7d43bd47a93-kube-api-access-pbwhg\") pod \"keda-admission-cf49989db-7djgx\" (UID: \"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93\") " pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.826340 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.826307 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:53.982052 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:53.980565 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-7djgx"] May 20 13:53:54.904500 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:54.904455 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-7djgx" event={"ID":"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93","Type":"ContainerStarted","Data":"61e3412d07e3ec7d53840db4b957a68c93e8b8f9433911fd0b765ab61d2758f8"} May 20 13:53:56.912729 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:56.912685 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-7djgx" event={"ID":"16d294da-b1d4-4e2c-a6bd-d7d43bd47a93","Type":"ContainerStarted","Data":"07ef27010bbb6997fbe41c9ddf528780a64ae5e819b505ac7620f41608d23fe6"} May 20 13:53:56.913243 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:56.912811 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:53:56.929546 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:53:56.929498 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-7djgx" podStartSLOduration=1.15438599 podStartE2EDuration="3.929481002s" podCreationTimestamp="2026-05-20 13:53:53 +0000 UTC" firstStartedPulling="2026-05-20 13:53:53.98338093 +0000 UTC m=+229.689812970" lastFinishedPulling="2026-05-20 13:53:56.75847594 +0000 UTC m=+232.464907982" observedRunningTime="2026-05-20 13:53:56.928338164 +0000 UTC m=+232.634770229" watchObservedRunningTime="2026-05-20 13:53:56.929481002 +0000 UTC m=+232.635913063" May 20 13:54:17.920598 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:54:17.920563 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-7djgx" May 20 13:55:04.818559 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:55:04.818529 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:55:04.819074 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:55:04.818539 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 13:56:26.586914 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.586805 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-98dxz"] May 20 13:56:26.590103 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.590081 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-98dxz" May 20 13:56:26.592626 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.592599 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 20 13:56:26.592750 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.592691 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 20 13:56:26.592750 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.592696 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" May 20 13:56:26.593480 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.593466 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cb4f8\"" May 20 13:56:26.597788 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.597768 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-98dxz"] May 20 13:56:26.683858 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.683799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.684025 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.683940 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz65\" (UniqueName: \"kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.784923 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.784885 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.785132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.784978 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz65\" (UniqueName: \"kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.785280 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.785259 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.793664 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.793632 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz65\" (UniqueName: \"kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65\") pod \"s3-init-98dxz\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " pod="kserve/s3-init-98dxz" May 20 13:56:26.910389 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:26.910302 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-98dxz" May 20 13:56:27.043540 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:27.043517 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-98dxz"] May 20 13:56:27.046811 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:56:27.046780 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d0f2895_540e_4d6b_a200_4f03751281d7.slice/crio-9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b WatchSource:0}: Error finding container 9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b: Status 404 returned error can't find the container with id 9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b May 20 13:56:27.049366 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:27.049345 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 13:56:27.359738 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:27.359705 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerStarted","Data":"9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b"} May 20 13:56:31.386519 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:31.385346 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerStarted","Data":"eaecb43887a125e1ab86eddaebbeda608ca07bd39ef49e71f15e7131a8eb0c43"} May 20 13:56:32.396452 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:32.396414 2554 generic.go:358] "Generic (PLEG): container finished" podID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerID="eaecb43887a125e1ab86eddaebbeda608ca07bd39ef49e71f15e7131a8eb0c43" exitCode=0 May 20 13:56:32.396957 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:32.396520 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerDied","Data":"eaecb43887a125e1ab86eddaebbeda608ca07bd39ef49e71f15e7131a8eb0c43"} May 20 13:56:37.417817 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:37.417783 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerStarted","Data":"3370b1e267a3d3579fda298ad3ca00095b941b44fa8663caaf90d7289097c6f4"} May 20 13:56:37.435593 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:56:37.435537 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-98dxz" podStartSLOduration=1.667038767 podStartE2EDuration="11.435518138s" podCreationTimestamp="2026-05-20 13:56:26 +0000 UTC" firstStartedPulling="2026-05-20 13:56:27.04953083 +0000 UTC m=+382.755962870" lastFinishedPulling="2026-05-20 13:56:36.8180102 +0000 UTC m=+392.524442241" observedRunningTime="2026-05-20 13:56:37.433343786 +0000 UTC m=+393.139775841" watchObservedRunningTime="2026-05-20 13:56:37.435518138 +0000 UTC m=+393.141950206" May 20 13:57:42.621579 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:42.621534 2554 generic.go:358] "Generic (PLEG): container finished" podID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerID="3370b1e267a3d3579fda298ad3ca00095b941b44fa8663caaf90d7289097c6f4" exitCode=0 May 20 13:57:42.622058 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:42.621606 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerDied","Data":"3370b1e267a3d3579fda298ad3ca00095b941b44fa8663caaf90d7289097c6f4"} May 20 13:57:43.749146 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.749121 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-98dxz" May 20 13:57:43.863633 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.863596 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache\") pod \"5d0f2895-540e-4d6b-a200-4f03751281d7\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " May 20 13:57:43.863810 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.863653 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqz65\" (UniqueName: \"kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65\") pod \"5d0f2895-540e-4d6b-a200-4f03751281d7\" (UID: \"5d0f2895-540e-4d6b-a200-4f03751281d7\") " May 20 13:57:43.866063 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.866027 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65" (OuterVolumeSpecName: "kube-api-access-cqz65") pod "5d0f2895-540e-4d6b-a200-4f03751281d7" (UID: "5d0f2895-540e-4d6b-a200-4f03751281d7"). InnerVolumeSpecName "kube-api-access-cqz65". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:57:43.882614 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.882541 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d0f2895-540e-4d6b-a200-4f03751281d7" (UID: "5d0f2895-540e-4d6b-a200-4f03751281d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:57:43.964400 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.964361 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0f2895-540e-4d6b-a200-4f03751281d7-model-cache\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:57:43.964400 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:43.964393 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz65\" (UniqueName: \"kubernetes.io/projected/5d0f2895-540e-4d6b-a200-4f03751281d7-kube-api-access-cqz65\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:57:44.628699 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:44.628664 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-98dxz" May 20 13:57:44.628699 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:44.628682 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-98dxz" event={"ID":"5d0f2895-540e-4d6b-a200-4f03751281d7","Type":"ContainerDied","Data":"9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b"} May 20 13:57:44.628699 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:44.628709 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a463b7a3b66aacf90582906777a73ef9551b9f1447d662f79a958f0f1f27f6b" May 20 13:57:54.155749 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.155713 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 13:57:54.156241 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.156101 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerName="s3-init" May 20 13:57:54.156241 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.156114 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerName="s3-init" May 20 13:57:54.156241 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.156136 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerName="download-hf-model" May 20 13:57:54.156241 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.156141 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerName="download-hf-model" May 20 13:57:54.156241 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.156189 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d0f2895-540e-4d6b-a200-4f03751281d7" containerName="s3-init" May 20 13:57:54.159428 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.159408 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.162078 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162047 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0beca-predictor-serving-cert\"" May 20 13:57:54.162223 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162109 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" May 20 13:57:54.162223 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162153 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 20 13:57:54.162223 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162201 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" May 20 13:57:54.162372 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162295 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-25znm\"" May 20 13:57:54.162372 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.162350 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 20 13:57:54.163077 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.163059 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\"" May 20 13:57:54.170278 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.170253 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 13:57:54.255243 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.255204 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq46q\" (UniqueName: \"kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.255450 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.255250 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.255450 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.255340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.255450 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.255390 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.255450 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.255441 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356310 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356270 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356494 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356320 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356494 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356372 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356494 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356432 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq46q\" (UniqueName: \"kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356494 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356466 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.356664 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:57:54.356528 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-0beca-predictor-serving-cert" not found May 20 13:57:54.356664 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:57:54.356595 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls podName:46a9250f-c371-4271-8628-aba0317bb7d1 nodeName:}" failed. No retries permitted until 2026-05-20 13:57:54.856574398 +0000 UTC m=+470.563006453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls") pod "isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1") : secret "isvc-raw-sklearn-batcher-0beca-predictor-serving-cert" not found May 20 13:57:54.356891 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356821 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.357021 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.356972 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.357072 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.357021 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.365175 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.365149 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq46q\" (UniqueName: \"kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.860428 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.860390 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:54.863027 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:54.862995 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:55.071354 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:55.071319 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:57:55.208755 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:55.208710 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 13:57:55.211205 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:57:55.211182 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a9250f_c371_4271_8628_aba0317bb7d1.slice/crio-334c2c35ea1d47874258d618bb6320bc097c86187481f58646f262b2eed42bf1 WatchSource:0}: Error finding container 334c2c35ea1d47874258d618bb6320bc097c86187481f58646f262b2eed42bf1: Status 404 returned error can't find the container with id 334c2c35ea1d47874258d618bb6320bc097c86187481f58646f262b2eed42bf1 May 20 13:57:55.663734 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:55.663689 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerStarted","Data":"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a"} May 20 13:57:55.663734 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:55.663739 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerStarted","Data":"334c2c35ea1d47874258d618bb6320bc097c86187481f58646f262b2eed42bf1"} May 20 13:57:59.679753 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:59.679714 2554 generic.go:358] "Generic (PLEG): container finished" podID="46a9250f-c371-4271-8628-aba0317bb7d1" containerID="17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a" exitCode=0 May 20 13:57:59.680173 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:57:59.679790 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerDied","Data":"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a"} May 20 13:58:12.728360 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:12.728320 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerStarted","Data":"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024"} May 20 13:58:14.737256 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:14.737216 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerStarted","Data":"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769"} May 20 13:58:17.751019 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:17.750981 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerStarted","Data":"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581"} May 20 13:58:17.751429 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:17.751106 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:58:17.775081 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:17.775028 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podStartSLOduration=6.027178227 podStartE2EDuration="23.775011662s" podCreationTimestamp="2026-05-20 13:57:54 +0000 UTC" firstStartedPulling="2026-05-20 13:57:59.680937711 +0000 UTC m=+475.387369753" lastFinishedPulling="2026-05-20 13:58:17.428771143 +0000 UTC m=+493.135203188" observedRunningTime="2026-05-20 13:58:17.772892975 +0000 UTC m=+493.479325040" watchObservedRunningTime="2026-05-20 13:58:17.775011662 +0000 UTC m=+493.481443723" May 20 13:58:18.754539 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:18.754501 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:58:18.754539 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:18.754540 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:58:18.755789 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:18.755751 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:18.756450 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:18.756429 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:58:19.757820 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:19.757779 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:19.758382 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:19.758356 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:58:19.761650 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:19.761630 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:58:20.766291 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:20.766254 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:20.766760 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:20.766590 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:58:30.766829 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:30.766774 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:30.767303 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:30.767204 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:58:40.766199 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:40.766146 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:40.766688 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:40.766665 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:58:50.766416 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:50.766359 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:58:50.766905 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:58:50.766879 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:00.766876 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:00.766747 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:59:00.767257 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:00.767206 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:10.766779 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:10.766727 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:59:10.767311 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:10.767218 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:20.767120 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:20.767082 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:20.767506 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:20.767176 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:29.148898 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.148861 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 13:59:29.149415 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.149364 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" containerID="cri-o://3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024" gracePeriod=30 May 20 13:59:29.149533 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.149443 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" containerID="cri-o://24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581" gracePeriod=30 May 20 13:59:29.149617 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.149464 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" containerID="cri-o://05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769" gracePeriod=30 May 20 13:59:29.258007 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.257971 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 13:59:29.262132 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.261905 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.264290 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.264265 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-4de28-predictor-serving-cert\"" May 20 13:59:29.264432 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.264292 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\"" May 20 13:59:29.274791 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.274768 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 13:59:29.332937 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.332905 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 13:59:29.336502 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.336480 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.338820 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.338800 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4de28-predictor-serving-cert\"" May 20 13:59:29.338958 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.338889 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\"" May 20 13:59:29.346108 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.346075 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 13:59:29.426642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426546 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.426642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426590 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.426642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.426642 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426639 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79df\" (UniqueName: \"kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426708 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426727 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426746 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426769 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.427008 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.426787 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtchb\" (UniqueName: \"kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.527931 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.527895 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.527931 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.527938 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.528205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.527969 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528014 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.528205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528041 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtchb\" (UniqueName: \"kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.528205 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528137 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.528396 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528303 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528396 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528349 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528486 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528391 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d79df\" (UniqueName: \"kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528486 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528420 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528586 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:59:29.528526 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-serving-cert: secret "isvc-sklearn-graph-raw-4de28-predictor-serving-cert" not found May 20 13:59:29.528637 ip-10-0-140-251 kubenswrapper[2554]: E0520 13:59:29.528611 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls podName:29a375df-d7c7-4e26-8780-ee6ff78e9dcc nodeName:}" failed. No retries permitted until 2026-05-20 13:59:30.028591174 +0000 UTC m=+565.735023225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls") pod "isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc") : secret "isvc-sklearn-graph-raw-4de28-predictor-serving-cert" not found May 20 13:59:29.528698 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528674 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528765 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528698 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.528895 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528869 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.529034 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.528928 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.529125 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.529096 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.529220 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.529206 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.530891 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.530871 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.537483 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.537459 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79df\" (UniqueName: \"kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:29.537881 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.537862 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtchb\" (UniqueName: \"kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb\") pod \"isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.648009 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.647968 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:29.758594 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.758551 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:29.785013 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.784984 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 13:59:29.787375 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:59:29.787332 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095473d0_4ee6_42f5_bcf0_ff52bb8660ef.slice/crio-4178ac3d8c888d1af957f538941e4c12dd13ae1ce580dbad991b5363d403c53b WatchSource:0}: Error finding container 4178ac3d8c888d1af957f538941e4c12dd13ae1ce580dbad991b5363d403c53b: Status 404 returned error can't find the container with id 4178ac3d8c888d1af957f538941e4c12dd13ae1ce580dbad991b5363d403c53b May 20 13:59:29.979684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.979587 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerStarted","Data":"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5"} May 20 13:59:29.979684 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.979631 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerStarted","Data":"4178ac3d8c888d1af957f538941e4c12dd13ae1ce580dbad991b5363d403c53b"} May 20 13:59:29.985134 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.985097 2554 generic.go:358] "Generic (PLEG): container finished" podID="46a9250f-c371-4271-8628-aba0317bb7d1" containerID="05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769" exitCode=2 May 20 13:59:29.985281 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:29.985161 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerDied","Data":"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769"} May 20 13:59:30.033424 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.033381 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:30.035948 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.035929 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") pod \"isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:30.173359 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.173324 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:30.312920 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.312883 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 13:59:30.316175 ip-10-0-140-251 kubenswrapper[2554]: W0520 13:59:30.316144 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a375df_d7c7_4e26_8780_ee6ff78e9dcc.slice/crio-d4c51b07aa42fa0ea5001e8182bc0229ba7f85f5cc03a1a9f5b3b2b3debd7c77 WatchSource:0}: Error finding container d4c51b07aa42fa0ea5001e8182bc0229ba7f85f5cc03a1a9f5b3b2b3debd7c77: Status 404 returned error can't find the container with id d4c51b07aa42fa0ea5001e8182bc0229ba7f85f5cc03a1a9f5b3b2b3debd7c77 May 20 13:59:30.766214 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.766165 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:59:30.766526 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.766506 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:30.995307 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.995257 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerStarted","Data":"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853"} May 20 13:59:30.995307 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:30.995307 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerStarted","Data":"d4c51b07aa42fa0ea5001e8182bc0229ba7f85f5cc03a1a9f5b3b2b3debd7c77"} May 20 13:59:34.005503 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.005465 2554 generic.go:358] "Generic (PLEG): container finished" podID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerID="9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853" exitCode=0 May 20 13:59:34.005950 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.005543 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerDied","Data":"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853"} May 20 13:59:34.007613 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.007591 2554 generic.go:358] "Generic (PLEG): container finished" podID="46a9250f-c371-4271-8628-aba0317bb7d1" containerID="3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024" exitCode=0 May 20 13:59:34.007706 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.007655 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerDied","Data":"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024"} May 20 13:59:34.009091 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.009067 2554 generic.go:358] "Generic (PLEG): container finished" podID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerID="7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5" exitCode=0 May 20 13:59:34.009171 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.009145 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerDied","Data":"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5"} May 20 13:59:34.759133 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:34.759051 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:35.015877 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.015774 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerStarted","Data":"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8"} May 20 13:59:35.015877 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.015818 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerStarted","Data":"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85"} May 20 13:59:35.016498 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.016233 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:35.016498 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.016262 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:35.017738 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.017701 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 13:59:35.038774 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:35.038061 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podStartSLOduration=6.038042496 podStartE2EDuration="6.038042496s" podCreationTimestamp="2026-05-20 13:59:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:59:35.034891893 +0000 UTC m=+570.741323957" watchObservedRunningTime="2026-05-20 13:59:35.038042496 +0000 UTC m=+570.744474560" May 20 13:59:36.020024 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:36.019978 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 13:59:39.758395 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:39.758350 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:39.758872 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:39.758503 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:40.767065 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:40.767000 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:59:40.767519 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:40.767427 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:41.025366 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:41.025286 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 13:59:41.026034 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:41.026001 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 13:59:44.758404 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:44.758355 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:49.758150 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:49.758095 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:50.766432 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:50.766379 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" May 20 13:59:50.766930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:50.766563 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:50.766930 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:50.766815 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 13:59:50.767061 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:50.766959 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:51.026395 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:51.026312 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 13:59:54.759038 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:54.758929 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" May 20 13:59:55.093314 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:55.093227 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerStarted","Data":"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825"} May 20 13:59:55.093314 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:55.093271 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerStarted","Data":"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7"} May 20 13:59:55.093573 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:55.093552 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:55.113180 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:55.113119 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podStartSLOduration=5.635966557 podStartE2EDuration="26.113097504s" podCreationTimestamp="2026-05-20 13:59:29 +0000 UTC" firstStartedPulling="2026-05-20 13:59:34.010250369 +0000 UTC m=+569.716682409" lastFinishedPulling="2026-05-20 13:59:54.487381315 +0000 UTC m=+590.193813356" observedRunningTime="2026-05-20 13:59:55.111187251 +0000 UTC m=+590.817619313" watchObservedRunningTime="2026-05-20 13:59:55.113097504 +0000 UTC m=+590.819529566" May 20 13:59:56.096712 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:56.096677 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 13:59:56.097853 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:56.097804 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 13:59:57.099952 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:57.099907 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 13:59:59.301922 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.301893 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 13:59:59.389197 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389166 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location\") pod \"46a9250f-c371-4271-8628-aba0317bb7d1\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " May 20 13:59:59.389398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389230 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\") pod \"46a9250f-c371-4271-8628-aba0317bb7d1\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " May 20 13:59:59.389398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389276 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") pod \"46a9250f-c371-4271-8628-aba0317bb7d1\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " May 20 13:59:59.389398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389306 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq46q\" (UniqueName: \"kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q\") pod \"46a9250f-c371-4271-8628-aba0317bb7d1\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " May 20 13:59:59.389398 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389334 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert\") pod \"46a9250f-c371-4271-8628-aba0317bb7d1\" (UID: \"46a9250f-c371-4271-8628-aba0317bb7d1\") " May 20 13:59:59.389659 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389634 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config") pod "46a9250f-c371-4271-8628-aba0317bb7d1" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:59:59.389726 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389591 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46a9250f-c371-4271-8628-aba0317bb7d1" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 13:59:59.389763 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.389734 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46a9250f-c371-4271-8628-aba0317bb7d1" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 13:59:59.391733 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.391703 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46a9250f-c371-4271-8628-aba0317bb7d1" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 13:59:59.391891 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.391727 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q" (OuterVolumeSpecName: "kube-api-access-vq46q") pod "46a9250f-c371-4271-8628-aba0317bb7d1" (UID: "46a9250f-c371-4271-8628-aba0317bb7d1"). InnerVolumeSpecName "kube-api-access-vq46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 13:59:59.490346 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.490309 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:59:59.490346 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.490340 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46a9250f-c371-4271-8628-aba0317bb7d1-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:59:59.490564 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.490355 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46a9250f-c371-4271-8628-aba0317bb7d1-isvc-raw-sklearn-batcher-0beca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:59:59.490564 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.490370 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46a9250f-c371-4271-8628-aba0317bb7d1-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 13:59:59.490564 ip-10-0-140-251 kubenswrapper[2554]: I0520 13:59:59.490383 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq46q\" (UniqueName: \"kubernetes.io/projected/46a9250f-c371-4271-8628-aba0317bb7d1-kube-api-access-vq46q\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:00:00.112735 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.112697 2554 generic.go:358] "Generic (PLEG): container finished" podID="46a9250f-c371-4271-8628-aba0317bb7d1" containerID="24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581" exitCode=0 May 20 14:00:00.112923 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.112773 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerDied","Data":"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581"} May 20 14:00:00.112923 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.112802 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" May 20 14:00:00.112923 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.112814 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt" event={"ID":"46a9250f-c371-4271-8628-aba0317bb7d1","Type":"ContainerDied","Data":"334c2c35ea1d47874258d618bb6320bc097c86187481f58646f262b2eed42bf1"} May 20 14:00:00.112923 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.112851 2554 scope.go:117] "RemoveContainer" containerID="24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581" May 20 14:00:00.121429 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.121410 2554 scope.go:117] "RemoveContainer" containerID="05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769" May 20 14:00:00.128677 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.128661 2554 scope.go:117] "RemoveContainer" containerID="3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024" May 20 14:00:00.136345 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.136322 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 14:00:00.136648 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.136630 2554 scope.go:117] "RemoveContainer" containerID="17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a" May 20 14:00:00.144361 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.144345 2554 scope.go:117] "RemoveContainer" containerID="24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581" May 20 14:00:00.144655 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:00:00.144633 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581\": container with ID starting with 24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581 not found: ID does not exist" containerID="24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581" May 20 14:00:00.144723 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.144669 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581"} err="failed to get container status \"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581\": rpc error: code = NotFound desc = could not find container \"24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581\": container with ID starting with 24ad182643d4099955ad43e21a65c7989d698cc39fe1d218dd0f686ea9bc6581 not found: ID does not exist" May 20 14:00:00.144723 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.144695 2554 scope.go:117] "RemoveContainer" containerID="05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769" May 20 14:00:00.145192 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:00:00.145057 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769\": container with ID starting with 05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769 not found: ID does not exist" containerID="05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769" May 20 14:00:00.145192 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.145090 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769"} err="failed to get container status \"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769\": rpc error: code = NotFound desc = could not find container \"05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769\": container with ID starting with 05b4d35e8374b5cde9a0fc354360381874b7efbd2b0a304ca4d5cc39199ec769 not found: ID does not exist" May 20 14:00:00.145192 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.145122 2554 scope.go:117] "RemoveContainer" containerID="3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024" May 20 14:00:00.145979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.145958 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0beca-predictor-6bdcf7c4dc-zjkzt"] May 20 14:00:00.145979 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:00:00.145967 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024\": container with ID starting with 3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024 not found: ID does not exist" containerID="3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024" May 20 14:00:00.146086 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.145991 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024"} err="failed to get container status \"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024\": rpc error: code = NotFound desc = could not find container \"3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024\": container with ID starting with 3765de796abd6e137843832d4fa3c22a95eac10339bb9092f838a7964a1ac024 not found: ID does not exist" May 20 14:00:00.146086 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.146012 2554 scope.go:117] "RemoveContainer" containerID="17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a" May 20 14:00:00.146264 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:00:00.146248 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a\": container with ID starting with 17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a not found: ID does not exist" containerID="17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a" May 20 14:00:00.146307 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.146266 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a"} err="failed to get container status \"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a\": rpc error: code = NotFound desc = could not find container \"17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a\": container with ID starting with 17572a9badf895337ea20a732e6fc36ba2d2d016543eb2f899e1d4ac40027a2a not found: ID does not exist" May 20 14:00:00.928509 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:00.928470 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" path="/var/lib/kubelet/pods/46a9250f-c371-4271-8628-aba0317bb7d1/volumes" May 20 14:00:01.026583 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:01.026541 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:00:02.104708 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:02.104674 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 14:00:02.105396 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:02.105363 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:00:04.852440 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:04.852408 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:00:04.853106 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:04.853083 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:00:11.026779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:11.026737 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:00:12.105553 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:12.105505 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:00:21.026139 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:21.026097 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:00:22.105872 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:22.105768 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:00:31.026280 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:31.026242 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:00:32.106207 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:32.106159 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:00:41.026744 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:41.026698 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:00:42.105724 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:42.105682 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:00:51.026550 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:51.026516 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 14:00:52.105806 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:00:52.105764 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:01:02.105939 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:02.105910 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 14:01:19.498379 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.498339 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 14:01:19.498914 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.498671 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" containerID="cri-o://53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85" gracePeriod=30 May 20 14:01:19.498914 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.498696 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kube-rbac-proxy" containerID="cri-o://17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8" gracePeriod=30 May 20 14:01:19.567566 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.567534 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:01:19.567986 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.567971 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.567987 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.567999 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568005 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568015 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568020 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568038 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="storage-initializer" May 20 14:01:19.568046 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568043 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="storage-initializer" May 20 14:01:19.568337 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568118 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="agent" May 20 14:01:19.568337 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568126 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kube-rbac-proxy" May 20 14:01:19.568337 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.568133 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="46a9250f-c371-4271-8628-aba0317bb7d1" containerName="kserve-container" May 20 14:01:19.570796 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.570779 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.574182 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.574160 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-de018-predictor-serving-cert\"" May 20 14:01:19.574182 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.574161 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\"" May 20 14:01:19.581886 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.581863 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:01:19.613899 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.613867 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qrk\" (UniqueName: \"kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.614053 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.613912 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.614053 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.613945 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.614053 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.614002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.614053 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.614036 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.661636 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.661604 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 14:01:19.662051 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.662014 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" containerID="cri-o://522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7" gracePeriod=30 May 20 14:01:19.662198 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.662072 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kube-rbac-proxy" containerID="cri-o://aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825" gracePeriod=30 May 20 14:01:19.714743 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.714705 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qrk\" (UniqueName: \"kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.714951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.714761 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.714951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.714809 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.714951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.714873 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.714951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.714900 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.715180 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:19.715037 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-de018-predictor-serving-cert" not found May 20 14:01:19.715180 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:19.715120 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls podName:2a73893f-48bb-4696-a9d7-8a63663df93d nodeName:}" failed. No retries permitted until 2026-05-20 14:01:20.215098576 +0000 UTC m=+675.921530616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d") : secret "isvc-sklearn-graph-raw-hpa-de018-predictor-serving-cert" not found May 20 14:01:19.715314 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.715288 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.715515 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.715495 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.715583 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.715552 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:19.723758 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:19.723736 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qrk\" (UniqueName: \"kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:20.014406 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.014368 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:01:20.021697 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.021666 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.024671 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.024600 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-de018-predictor-serving-cert\"" May 20 14:01:20.024890 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.024629 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\"" May 20 14:01:20.027174 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.027140 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:01:20.123786 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.123750 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.123979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.123806 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.123979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.123916 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.123979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.123959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.124139 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.124013 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwx4\" (UniqueName: \"kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225130 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225084 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:20.225322 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225141 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225322 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225215 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225322 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225268 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225322 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225305 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225533 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225354 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwx4\" (UniqueName: \"kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225533 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225500 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.225913 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.225894 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.226299 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.226276 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.227894 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.227870 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.227894 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.227884 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:20.233808 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.233785 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwx4\" (UniqueName: \"kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4\") pod \"isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.334729 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.334651 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:20.367356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.367322 2554 generic.go:358] "Generic (PLEG): container finished" podID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerID="aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825" exitCode=2 May 20 14:01:20.367526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.367398 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerDied","Data":"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825"} May 20 14:01:20.369436 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.369411 2554 generic.go:358] "Generic (PLEG): container finished" podID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerID="17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8" exitCode=2 May 20 14:01:20.369531 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.369452 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerDied","Data":"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8"} May 20 14:01:20.476738 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.476693 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:01:20.479117 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:01:20.479087 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fcc3b2_687c_4381_816d_dc512d05f4bb.slice/crio-3ab92fab4336c922d8aa9a754449b37336cd6deb2024fc24dc4d032539e3331e WatchSource:0}: Error finding container 3ab92fab4336c922d8aa9a754449b37336cd6deb2024fc24dc4d032539e3331e: Status 404 returned error can't find the container with id 3ab92fab4336c922d8aa9a754449b37336cd6deb2024fc24dc4d032539e3331e May 20 14:01:20.481196 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.481173 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:20.623747 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:20.623723 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:01:20.626909 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:01:20.626881 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a73893f_48bb_4696_a9d7_8a63663df93d.slice/crio-8bb052831603290a626497cc395ec9571812cd0862b2d07dae28a34962f7ebd4 WatchSource:0}: Error finding container 8bb052831603290a626497cc395ec9571812cd0862b2d07dae28a34962f7ebd4: Status 404 returned error can't find the container with id 8bb052831603290a626497cc395ec9571812cd0862b2d07dae28a34962f7ebd4 May 20 14:01:21.020823 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.020768 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused" May 20 14:01:21.026052 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.026015 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" May 20 14:01:21.374797 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.374702 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerStarted","Data":"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784"} May 20 14:01:21.374797 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.374748 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerStarted","Data":"3ab92fab4336c922d8aa9a754449b37336cd6deb2024fc24dc4d032539e3331e"} May 20 14:01:21.376085 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.376061 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerStarted","Data":"f47debd1e6d48ce839d758be361196e2d0975c7fcad4e30d3ac4338431adddbb"} May 20 14:01:21.376085 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:21.376087 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerStarted","Data":"8bb052831603290a626497cc395ec9571812cd0862b2d07dae28a34962f7ebd4"} May 20 14:01:22.100724 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:22.100683 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused" May 20 14:01:22.105915 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:22.105878 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" May 20 14:01:23.600576 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.600553 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 14:01:23.654340 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654308 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " May 20 14:01:23.654525 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654358 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtchb\" (UniqueName: \"kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb\") pod \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " May 20 14:01:23.654525 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654396 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert\") pod \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " May 20 14:01:23.654525 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654430 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls\") pod \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " May 20 14:01:23.654525 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654451 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location\") pod \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\" (UID: \"095473d0-4ee6-42f5-bcf0-ff52bb8660ef\") " May 20 14:01:23.654890 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654858 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "095473d0-4ee6-42f5-bcf0-ff52bb8660ef" (UID: "095473d0-4ee6-42f5-bcf0-ff52bb8660ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:01:23.654998 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654825 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config") pod "095473d0-4ee6-42f5-bcf0-ff52bb8660ef" (UID: "095473d0-4ee6-42f5-bcf0-ff52bb8660ef"). InnerVolumeSpecName "isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:01:23.654998 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.654896 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "095473d0-4ee6-42f5-bcf0-ff52bb8660ef" (UID: "095473d0-4ee6-42f5-bcf0-ff52bb8660ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:01:23.656745 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.656721 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "095473d0-4ee6-42f5-bcf0-ff52bb8660ef" (UID: "095473d0-4ee6-42f5-bcf0-ff52bb8660ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:01:23.656745 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.656737 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb" (OuterVolumeSpecName: "kube-api-access-wtchb") pod "095473d0-4ee6-42f5-bcf0-ff52bb8660ef" (UID: "095473d0-4ee6-42f5-bcf0-ff52bb8660ef"). InnerVolumeSpecName "kube-api-access-wtchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:01:23.755224 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.755188 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-isvc-xgboost-graph-raw-4de28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:23.755224 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.755219 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtchb\" (UniqueName: \"kubernetes.io/projected/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kube-api-access-wtchb\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:23.755224 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.755229 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:23.755449 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.755238 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:23.755449 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:23.755246 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/095473d0-4ee6-42f5-bcf0-ff52bb8660ef-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.140662 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.140639 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 14:01:24.260979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.260877 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert\") pod \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " May 20 14:01:24.260979 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.260937 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\") pod \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " May 20 14:01:24.261221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.260991 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") pod \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " May 20 14:01:24.261221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.261042 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d79df\" (UniqueName: \"kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df\") pod \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " May 20 14:01:24.261221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.261069 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location\") pod \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\" (UID: \"29a375df-d7c7-4e26-8780-ee6ff78e9dcc\") " May 20 14:01:24.261487 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.261437 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29a375df-d7c7-4e26-8780-ee6ff78e9dcc" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:01:24.261487 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.261456 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config") pod "29a375df-d7c7-4e26-8780-ee6ff78e9dcc" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc"). InnerVolumeSpecName "isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:01:24.261597 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.261519 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29a375df-d7c7-4e26-8780-ee6ff78e9dcc" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:01:24.263336 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.263308 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29a375df-d7c7-4e26-8780-ee6ff78e9dcc" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:01:24.263658 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.263635 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df" (OuterVolumeSpecName: "kube-api-access-d79df") pod "29a375df-d7c7-4e26-8780-ee6ff78e9dcc" (UID: "29a375df-d7c7-4e26-8780-ee6ff78e9dcc"). InnerVolumeSpecName "kube-api-access-d79df". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:01:24.362096 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.362058 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.362096 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.362090 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-isvc-sklearn-graph-raw-4de28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.362096 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.362103 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.362335 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.362112 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79df\" (UniqueName: \"kubernetes.io/projected/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kube-api-access-d79df\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.362335 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.362122 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a375df-d7c7-4e26-8780-ee6ff78e9dcc-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:01:24.390122 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.390091 2554 generic.go:358] "Generic (PLEG): container finished" podID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerID="ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784" exitCode=0 May 20 14:01:24.390297 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.390169 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerDied","Data":"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784"} May 20 14:01:24.392112 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.392089 2554 generic.go:358] "Generic (PLEG): container finished" podID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerID="522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7" exitCode=0 May 20 14:01:24.392237 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.392165 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" May 20 14:01:24.392237 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.392168 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerDied","Data":"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7"} May 20 14:01:24.392237 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.392198 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn" event={"ID":"095473d0-4ee6-42f5-bcf0-ff52bb8660ef","Type":"ContainerDied","Data":"4178ac3d8c888d1af957f538941e4c12dd13ae1ce580dbad991b5363d403c53b"} May 20 14:01:24.392237 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.392220 2554 scope.go:117] "RemoveContainer" containerID="aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825" May 20 14:01:24.403281 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.403241 2554 generic.go:358] "Generic (PLEG): container finished" podID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerID="53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85" exitCode=0 May 20 14:01:24.403386 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.403292 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerDied","Data":"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85"} May 20 14:01:24.403489 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.403474 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" event={"ID":"29a375df-d7c7-4e26-8780-ee6ff78e9dcc","Type":"ContainerDied","Data":"d4c51b07aa42fa0ea5001e8182bc0229ba7f85f5cc03a1a9f5b3b2b3debd7c77"} May 20 14:01:24.403531 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.403493 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd" May 20 14:01:24.487966 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.487802 2554 scope.go:117] "RemoveContainer" containerID="522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7" May 20 14:01:24.507777 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.507751 2554 scope.go:117] "RemoveContainer" containerID="7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5" May 20 14:01:24.516320 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.516300 2554 scope.go:117] "RemoveContainer" containerID="aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825" May 20 14:01:24.516647 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.516627 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825\": container with ID starting with aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825 not found: ID does not exist" containerID="aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825" May 20 14:01:24.516698 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.516658 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825"} err="failed to get container status \"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825\": rpc error: code = NotFound desc = could not find container \"aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825\": container with ID starting with aa08de7523596c4e2c2f6915ac02a7aab0ca55a2db9b81ca687c87ca2a4e7825 not found: ID does not exist" May 20 14:01:24.516698 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.516677 2554 scope.go:117] "RemoveContainer" containerID="522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7" May 20 14:01:24.516946 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.516928 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7\": container with ID starting with 522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7 not found: ID does not exist" containerID="522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7" May 20 14:01:24.517001 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.516950 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7"} err="failed to get container status \"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7\": rpc error: code = NotFound desc = could not find container \"522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7\": container with ID starting with 522ee5100a1a81835dde00dbe326e7825799c19210f6f8bef846d6c45ada91f7 not found: ID does not exist" May 20 14:01:24.517001 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.516963 2554 scope.go:117] "RemoveContainer" containerID="7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5" May 20 14:01:24.517210 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.517194 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5\": container with ID starting with 7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5 not found: ID does not exist" containerID="7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5" May 20 14:01:24.517259 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.517215 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5"} err="failed to get container status \"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5\": rpc error: code = NotFound desc = could not find container \"7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5\": container with ID starting with 7daf60c0a828e9d57cad7ed882c429484507a20af573c78e8030f60ebedd91e5 not found: ID does not exist" May 20 14:01:24.517259 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.517230 2554 scope.go:117] "RemoveContainer" containerID="17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8" May 20 14:01:24.524739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.524721 2554 scope.go:117] "RemoveContainer" containerID="53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85" May 20 14:01:24.531332 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.531306 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 14:01:24.533387 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.533359 2554 scope.go:117] "RemoveContainer" containerID="9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853" May 20 14:01:24.538154 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.538130 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4de28-predictor-76576dd44f-vcjhn"] May 20 14:01:24.541153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541089 2554 scope.go:117] "RemoveContainer" containerID="17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8" May 20 14:01:24.541369 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.541350 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8\": container with ID starting with 17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8 not found: ID does not exist" containerID="17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8" May 20 14:01:24.541418 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541376 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8"} err="failed to get container status \"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8\": rpc error: code = NotFound desc = could not find container \"17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8\": container with ID starting with 17491cee796b93c30cf143e75a1f2ddf3a1bcb80b127ec4a2e6e8e66abba6ae8 not found: ID does not exist" May 20 14:01:24.541418 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541399 2554 scope.go:117] "RemoveContainer" containerID="53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85" May 20 14:01:24.541631 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.541615 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85\": container with ID starting with 53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85 not found: ID does not exist" containerID="53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85" May 20 14:01:24.541668 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541637 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85"} err="failed to get container status \"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85\": rpc error: code = NotFound desc = could not find container \"53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85\": container with ID starting with 53d5bfc2700d0a8c66b720b783d9b1c881354a5b80d313ccbeb02cd16f18ad85 not found: ID does not exist" May 20 14:01:24.541668 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541653 2554 scope.go:117] "RemoveContainer" containerID="9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853" May 20 14:01:24.541981 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:01:24.541965 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853\": container with ID starting with 9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853 not found: ID does not exist" containerID="9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853" May 20 14:01:24.542034 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.541985 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853"} err="failed to get container status \"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853\": rpc error: code = NotFound desc = could not find container \"9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853\": container with ID starting with 9f976e1799ca1289222edc7d4f5c5b37affe978092709e70533baa8eda5b2853 not found: ID does not exist" May 20 14:01:24.556462 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.556426 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 14:01:24.558594 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.558569 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4de28-predictor-8754c99b7-cshkd"] May 20 14:01:24.926829 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.926752 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" path="/var/lib/kubelet/pods/095473d0-4ee6-42f5-bcf0-ff52bb8660ef/volumes" May 20 14:01:24.927294 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:24.927280 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" path="/var/lib/kubelet/pods/29a375df-d7c7-4e26-8780-ee6ff78e9dcc/volumes" May 20 14:01:25.410210 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.410176 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerStarted","Data":"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35"} May 20 14:01:25.410492 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.410220 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerStarted","Data":"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee"} May 20 14:01:25.410568 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.410545 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:25.410706 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.410683 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:25.412266 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.412240 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:01:25.412533 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.412516 2554 generic.go:358] "Generic (PLEG): container finished" podID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerID="f47debd1e6d48ce839d758be361196e2d0975c7fcad4e30d3ac4338431adddbb" exitCode=0 May 20 14:01:25.412587 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.412544 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerDied","Data":"f47debd1e6d48ce839d758be361196e2d0975c7fcad4e30d3ac4338431adddbb"} May 20 14:01:25.434486 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:25.434438 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podStartSLOduration=6.434420132 podStartE2EDuration="6.434420132s" podCreationTimestamp="2026-05-20 14:01:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:01:25.431788988 +0000 UTC m=+681.138221049" watchObservedRunningTime="2026-05-20 14:01:25.434420132 +0000 UTC m=+681.140852196" May 20 14:01:26.419006 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.418967 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerStarted","Data":"cc014d5d493aaae821302099333d7bd3e496d27cce573de4cf16376f8fdc4d1a"} May 20 14:01:26.419006 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.419011 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerStarted","Data":"d7719941217ba9597c48b21289882d9412dcd73cee691bc1f47def7ed88b5d3e"} May 20 14:01:26.419435 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.419373 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:01:26.419435 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.419411 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:26.419549 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.419530 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:26.420759 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.420735 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:01:26.444180 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:26.444128 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podStartSLOduration=7.444114121 podStartE2EDuration="7.444114121s" podCreationTimestamp="2026-05-20 14:01:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:01:26.443413792 +0000 UTC m=+682.149845855" watchObservedRunningTime="2026-05-20 14:01:26.444114121 +0000 UTC m=+682.150546182" May 20 14:01:27.422799 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:27.422754 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:01:31.424225 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:31.424197 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:01:31.424808 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:31.424781 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:01:32.426899 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:32.426873 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:01:32.427468 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:32.427443 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:01:41.425506 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:41.425468 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:01:42.427482 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:42.427440 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:01:51.425590 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:51.425544 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:01:52.427506 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:01:52.427422 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:02:01.425303 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:01.425260 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:02:02.427707 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:02.427665 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:02:11.424715 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:11.424677 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:02:12.427481 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:12.427435 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:02:21.425581 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:21.425536 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:02:22.428248 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:22.428204 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:02:31.426035 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:31.425996 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:02:32.428895 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:32.428865 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:02:59.775310 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.775272 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:02:59.775931 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.775652 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" containerID="cri-o://d7719941217ba9597c48b21289882d9412dcd73cee691bc1f47def7ed88b5d3e" gracePeriod=30 May 20 14:02:59.775931 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.775693 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kube-rbac-proxy" containerID="cri-o://cc014d5d493aaae821302099333d7bd3e496d27cce573de4cf16376f8fdc4d1a" gracePeriod=30 May 20 14:02:59.812760 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.812720 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:02:59.813138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813125 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="storage-initializer" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813140 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="storage-initializer" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813150 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813155 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813167 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kube-rbac-proxy" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813173 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kube-rbac-proxy" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813181 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" May 20 14:02:59.813186 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813186 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813194 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kube-rbac-proxy" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813199 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kube-rbac-proxy" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813214 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="storage-initializer" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813220 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="storage-initializer" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813272 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kube-rbac-proxy" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813281 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="095473d0-4ee6-42f5-bcf0-ff52bb8660ef" containerName="kserve-container" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813287 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kserve-container" May 20 14:02:59.813390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.813293 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="29a375df-d7c7-4e26-8780-ee6ff78e9dcc" containerName="kube-rbac-proxy" May 20 14:02:59.816379 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.816359 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:02:59.818648 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.818626 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cbb25-predictor-serving-cert\"" May 20 14:02:59.818734 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.818637 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\"" May 20 14:02:59.827157 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.827126 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:02:59.854350 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.854313 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:02:59.854664 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.854619 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" containerID="cri-o://d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee" gracePeriod=30 May 20 14:02:59.854723 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.854661 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kube-rbac-proxy" containerID="cri-o://a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35" gracePeriod=30 May 20 14:02:59.905925 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.905888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:02:59.906101 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.905934 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:02:59.906101 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:02:59.905984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknv2\" (UniqueName: \"kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.007334 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.007289 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.007550 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.007372 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.007550 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.007496 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hknv2\" (UniqueName: \"kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.007790 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:00.007763 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-serving-cert: secret "message-dumper-raw-cbb25-predictor-serving-cert" not found May 20 14:03:00.007992 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:00.007976 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls podName:9cae34c1-3cb9-452b-9ec4-3cbd4077479a nodeName:}" failed. No retries permitted until 2026-05-20 14:03:00.507955435 +0000 UTC m=+776.214387491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls") pod "message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" (UID: "9cae34c1-3cb9-452b-9ec4-3cbd4077479a") : secret "message-dumper-raw-cbb25-predictor-serving-cert" not found May 20 14:03:00.008108 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.008082 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.016419 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.016390 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknv2\" (UniqueName: \"kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.512264 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.512223 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.514755 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.514724 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") pod \"message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.728632 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.728570 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:00.738818 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.738790 2554 generic.go:358] "Generic (PLEG): container finished" podID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerID="a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35" exitCode=2 May 20 14:03:00.738957 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.738890 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerDied","Data":"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35"} May 20 14:03:00.740825 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.740796 2554 generic.go:358] "Generic (PLEG): container finished" podID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerID="cc014d5d493aaae821302099333d7bd3e496d27cce573de4cf16376f8fdc4d1a" exitCode=2 May 20 14:03:00.740950 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.740868 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerDied","Data":"cc014d5d493aaae821302099333d7bd3e496d27cce573de4cf16376f8fdc4d1a"} May 20 14:03:00.863855 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.863777 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:03:00.866317 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:03:00.866290 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae34c1_3cb9_452b_9ec4_3cbd4077479a.slice/crio-a2531e57578c6fbb3deb69d910e24bcb44a337296168c78e2cf9065dd8f252e6 WatchSource:0}: Error finding container a2531e57578c6fbb3deb69d910e24bcb44a337296168c78e2cf9065dd8f252e6: Status 404 returned error can't find the container with id a2531e57578c6fbb3deb69d910e24bcb44a337296168c78e2cf9065dd8f252e6 May 20 14:03:00.868097 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:00.868077 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:03:01.420517 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:01.420470 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" May 20 14:03:01.425060 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:01.425016 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" May 20 14:03:01.746182 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:01.746136 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerStarted","Data":"a2531e57578c6fbb3deb69d910e24bcb44a337296168c78e2cf9065dd8f252e6"} May 20 14:03:02.424058 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.424002 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused" May 20 14:03:02.427677 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.427642 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" May 20 14:03:02.751138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.751104 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerStarted","Data":"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91"} May 20 14:03:02.751138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.751143 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerStarted","Data":"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4"} May 20 14:03:02.751433 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.751190 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:02.751433 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.751296 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:02.752987 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.752969 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:02.769430 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:02.769382 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" podStartSLOduration=2.838426952 podStartE2EDuration="3.76936775s" podCreationTimestamp="2026-05-20 14:02:59 +0000 UTC" firstStartedPulling="2026-05-20 14:03:00.86820759 +0000 UTC m=+776.574639631" lastFinishedPulling="2026-05-20 14:03:01.799148385 +0000 UTC m=+777.505580429" observedRunningTime="2026-05-20 14:03:02.767330409 +0000 UTC m=+778.473762472" watchObservedRunningTime="2026-05-20 14:03:02.76936775 +0000 UTC m=+778.475799888" May 20 14:03:03.906736 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.906713 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:03:03.944100 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944066 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls\") pod \"75fcc3b2-687c-4381-816d-dc512d05f4bb\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " May 20 14:03:03.944311 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944128 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location\") pod \"75fcc3b2-687c-4381-816d-dc512d05f4bb\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " May 20 14:03:03.944311 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944148 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert\") pod \"75fcc3b2-687c-4381-816d-dc512d05f4bb\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " May 20 14:03:03.944311 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944227 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"75fcc3b2-687c-4381-816d-dc512d05f4bb\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " May 20 14:03:03.944311 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944259 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkwx4\" (UniqueName: \"kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4\") pod \"75fcc3b2-687c-4381-816d-dc512d05f4bb\" (UID: \"75fcc3b2-687c-4381-816d-dc512d05f4bb\") " May 20 14:03:03.944607 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944578 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75fcc3b2-687c-4381-816d-dc512d05f4bb" (UID: "75fcc3b2-687c-4381-816d-dc512d05f4bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:03:03.944682 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944605 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75fcc3b2-687c-4381-816d-dc512d05f4bb" (UID: "75fcc3b2-687c-4381-816d-dc512d05f4bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:03.944682 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.944624 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config") pod "75fcc3b2-687c-4381-816d-dc512d05f4bb" (UID: "75fcc3b2-687c-4381-816d-dc512d05f4bb"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:03.946391 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.946367 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75fcc3b2-687c-4381-816d-dc512d05f4bb" (UID: "75fcc3b2-687c-4381-816d-dc512d05f4bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:03.946560 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:03.946544 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4" (OuterVolumeSpecName: "kube-api-access-pkwx4") pod "75fcc3b2-687c-4381-816d-dc512d05f4bb" (UID: "75fcc3b2-687c-4381-816d-dc512d05f4bb"). InnerVolumeSpecName "kube-api-access-pkwx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:04.045720 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.045633 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-isvc-xgboost-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:04.045720 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.045674 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwx4\" (UniqueName: \"kubernetes.io/projected/75fcc3b2-687c-4381-816d-dc512d05f4bb-kube-api-access-pkwx4\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:04.045720 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.045691 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75fcc3b2-687c-4381-816d-dc512d05f4bb-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:04.045720 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.045706 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75fcc3b2-687c-4381-816d-dc512d05f4bb-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:04.045981 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.045723 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75fcc3b2-687c-4381-816d-dc512d05f4bb-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:04.759274 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.759236 2554 generic.go:358] "Generic (PLEG): container finished" podID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerID="d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee" exitCode=0 May 20 14:03:04.759475 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.759288 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerDied","Data":"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee"} May 20 14:03:04.759475 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.759335 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" event={"ID":"75fcc3b2-687c-4381-816d-dc512d05f4bb","Type":"ContainerDied","Data":"3ab92fab4336c922d8aa9a754449b37336cd6deb2024fc24dc4d032539e3331e"} May 20 14:03:04.759475 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.759343 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6" May 20 14:03:04.759475 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.759356 2554 scope.go:117] "RemoveContainer" containerID="a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35" May 20 14:03:04.761951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.761919 2554 generic.go:358] "Generic (PLEG): container finished" podID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerID="d7719941217ba9597c48b21289882d9412dcd73cee691bc1f47def7ed88b5d3e" exitCode=0 May 20 14:03:04.762084 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.761957 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerDied","Data":"d7719941217ba9597c48b21289882d9412dcd73cee691bc1f47def7ed88b5d3e"} May 20 14:03:04.770760 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.770740 2554 scope.go:117] "RemoveContainer" containerID="d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee" May 20 14:03:04.780148 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.779921 2554 scope.go:117] "RemoveContainer" containerID="ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784" May 20 14:03:04.785339 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.785297 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:03:04.788010 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.787981 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-de018-predictor-5b84fff46f-g2xs6"] May 20 14:03:04.790400 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.790378 2554 scope.go:117] "RemoveContainer" containerID="a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35" May 20 14:03:04.790728 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:04.790704 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35\": container with ID starting with a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35 not found: ID does not exist" containerID="a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35" May 20 14:03:04.790795 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.790744 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35"} err="failed to get container status \"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35\": rpc error: code = NotFound desc = could not find container \"a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35\": container with ID starting with a3553b7d31b48b5b3f8784b216a4e734ed325d310a8297b82a7fc6fda7c4cf35 not found: ID does not exist" May 20 14:03:04.790795 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.790769 2554 scope.go:117] "RemoveContainer" containerID="d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee" May 20 14:03:04.791145 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:04.791124 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee\": container with ID starting with d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee not found: ID does not exist" containerID="d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee" May 20 14:03:04.791224 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.791153 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee"} err="failed to get container status \"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee\": rpc error: code = NotFound desc = could not find container \"d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee\": container with ID starting with d47a491d6dc1efc9c042ee618369ce671aa1f1fefbdd682c0c8df99a0c6603ee not found: ID does not exist" May 20 14:03:04.791224 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.791172 2554 scope.go:117] "RemoveContainer" containerID="ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784" May 20 14:03:04.791489 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:04.791467 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784\": container with ID starting with ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784 not found: ID does not exist" containerID="ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784" May 20 14:03:04.791540 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.791498 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784"} err="failed to get container status \"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784\": rpc error: code = NotFound desc = could not find container \"ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784\": container with ID starting with ebbda6a40a27740f89e1f788eed9739049ab3f48584c2932722c5d150d8a1784 not found: ID does not exist" May 20 14:03:04.831920 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.831895 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:03:04.932582 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.932544 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" path="/var/lib/kubelet/pods/75fcc3b2-687c-4381-816d-dc512d05f4bb/volumes" May 20 14:03:04.953951 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.953911 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4qrk\" (UniqueName: \"kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk\") pod \"2a73893f-48bb-4696-a9d7-8a63663df93d\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " May 20 14:03:04.954135 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.953967 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") pod \"2a73893f-48bb-4696-a9d7-8a63663df93d\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " May 20 14:03:04.954135 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.953996 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location\") pod \"2a73893f-48bb-4696-a9d7-8a63663df93d\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " May 20 14:03:04.954135 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.954078 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") pod \"2a73893f-48bb-4696-a9d7-8a63663df93d\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " May 20 14:03:04.954293 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.954137 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert\") pod \"2a73893f-48bb-4696-a9d7-8a63663df93d\" (UID: \"2a73893f-48bb-4696-a9d7-8a63663df93d\") " May 20 14:03:04.954436 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.954405 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a73893f-48bb-4696-a9d7-8a63663df93d" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:03:04.954507 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.954426 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config") pod "2a73893f-48bb-4696-a9d7-8a63663df93d" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:04.954507 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.954478 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a73893f-48bb-4696-a9d7-8a63663df93d" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:03:04.956237 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.956217 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a73893f-48bb-4696-a9d7-8a63663df93d" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:03:04.956305 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:04.956291 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk" (OuterVolumeSpecName: "kube-api-access-g4qrk") pod "2a73893f-48bb-4696-a9d7-8a63663df93d" (UID: "2a73893f-48bb-4696-a9d7-8a63663df93d"). InnerVolumeSpecName "kube-api-access-g4qrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:03:05.054800 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.054752 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qrk\" (UniqueName: \"kubernetes.io/projected/2a73893f-48bb-4696-a9d7-8a63663df93d-kube-api-access-g4qrk\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:05.054800 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.054793 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a73893f-48bb-4696-a9d7-8a63663df93d-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:05.054800 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.054804 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a73893f-48bb-4696-a9d7-8a63663df93d-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:05.055064 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.054814 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-isvc-sklearn-graph-raw-hpa-de018-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:05.055064 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.054826 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a73893f-48bb-4696-a9d7-8a63663df93d-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:03:05.767204 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.767173 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" event={"ID":"2a73893f-48bb-4696-a9d7-8a63663df93d","Type":"ContainerDied","Data":"8bb052831603290a626497cc395ec9571812cd0862b2d07dae28a34962f7ebd4"} May 20 14:03:05.767396 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.767222 2554 scope.go:117] "RemoveContainer" containerID="cc014d5d493aaae821302099333d7bd3e496d27cce573de4cf16376f8fdc4d1a" May 20 14:03:05.767396 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.767228 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd" May 20 14:03:05.775762 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.775738 2554 scope.go:117] "RemoveContainer" containerID="d7719941217ba9597c48b21289882d9412dcd73cee691bc1f47def7ed88b5d3e" May 20 14:03:05.783517 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.783359 2554 scope.go:117] "RemoveContainer" containerID="f47debd1e6d48ce839d758be361196e2d0975c7fcad4e30d3ac4338431adddbb" May 20 14:03:05.788810 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.788784 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:03:05.792373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:05.792346 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-de018-predictor-5fd94fb6f5-tvgmd"] May 20 14:03:06.927436 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:06.927400 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" path="/var/lib/kubelet/pods/2a73893f-48bb-4696-a9d7-8a63663df93d/volumes" May 20 14:03:09.766628 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:09.766597 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:03:19.859697 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.859656 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860035 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kube-rbac-proxy" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860048 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kube-rbac-proxy" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860061 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860066 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860075 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="storage-initializer" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860081 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="storage-initializer" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860088 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="storage-initializer" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860093 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="storage-initializer" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860099 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860103 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860120 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kube-rbac-proxy" May 20 14:03:19.860142 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860125 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kube-rbac-proxy" May 20 14:03:19.860547 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860173 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kube-rbac-proxy" May 20 14:03:19.860547 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860180 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kube-rbac-proxy" May 20 14:03:19.860547 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860186 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="75fcc3b2-687c-4381-816d-dc512d05f4bb" containerName="kserve-container" May 20 14:03:19.860547 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.860192 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a73893f-48bb-4696-a9d7-8a63663df93d" containerName="kserve-container" May 20 14:03:19.866473 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.866449 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:19.868756 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.868733 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cbb25-predictor-serving-cert\"" May 20 14:03:19.868915 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.868894 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\"" May 20 14:03:19.868974 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.868926 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" May 20 14:03:19.869226 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.869199 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" May 20 14:03:19.872739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.872711 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:03:19.982359 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.982313 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:19.982359 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.982361 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:19.982558 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.982443 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:19.982558 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.982500 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bwf\" (UniqueName: \"kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:19.982558 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:19.982552 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.085671 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.085618 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086324 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086403 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086464 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26bwf\" (UniqueName: \"kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086529 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:20.086582 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-serving-cert: secret "isvc-logger-raw-cbb25-predictor-serving-cert" not found May 20 14:03:20.086779 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:03:20.086652 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls podName:e2977e91-a2b4-4808-8558-4184e458a9b2 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:20.586632276 +0000 UTC m=+796.293064316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls") pod "isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2") : secret "isvc-logger-raw-cbb25-predictor-serving-cert" not found May 20 14:03:20.087136 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086863 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.087136 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086887 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.087136 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.086928 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.095050 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.095014 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bwf\" (UniqueName: \"kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.591024 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.590972 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.593598 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.593565 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") pod \"isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.778413 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.778376 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:20.918737 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:20.918708 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:03:20.921541 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:03:20.921509 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2977e91_a2b4_4808_8558_4184e458a9b2.slice/crio-615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db WatchSource:0}: Error finding container 615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db: Status 404 returned error can't find the container with id 615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db May 20 14:03:21.821572 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:21.821537 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerStarted","Data":"f68ad9e4ecbcb11d673267f826a9e6097fc089280e640385ac381397e2e3d1c3"} May 20 14:03:21.821572 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:21.821574 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerStarted","Data":"615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db"} May 20 14:03:24.832669 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:24.832636 2554 generic.go:358] "Generic (PLEG): container finished" podID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerID="f68ad9e4ecbcb11d673267f826a9e6097fc089280e640385ac381397e2e3d1c3" exitCode=0 May 20 14:03:24.833067 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:24.832719 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerDied","Data":"f68ad9e4ecbcb11d673267f826a9e6097fc089280e640385ac381397e2e3d1c3"} May 20 14:03:25.837859 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.837807 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerStarted","Data":"b05f01e8acb85ffae5fea5284ad2f4e6ebff036e57d36dc9e25dd1c695f64ab1"} May 20 14:03:25.837859 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.837864 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerStarted","Data":"ae5ac74c01dab31506a768a6db49fb6f2d00466fb6300ec0c613e22af56190c8"} May 20 14:03:25.838271 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.837874 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerStarted","Data":"9b0eb757ca78aff3220435a4af0eeba08729dc46702762cc5280f5ec7e60cfb3"} May 20 14:03:25.838271 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.838190 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:25.838330 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.838321 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:25.839641 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.839617 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:25.859254 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:25.859192 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podStartSLOduration=6.859174355 podStartE2EDuration="6.859174355s" podCreationTimestamp="2026-05-20 14:03:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:03:25.856894656 +0000 UTC m=+801.563326717" watchObservedRunningTime="2026-05-20 14:03:25.859174355 +0000 UTC m=+801.565606420" May 20 14:03:26.841027 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:26.840991 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:26.841504 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:26.841034 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:26.841980 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:26.841958 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:03:27.844758 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:27.844712 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:27.845317 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:27.845286 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:03:32.849180 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:32.849145 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:03:32.849861 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:32.849812 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:32.850134 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:32.850104 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:03:42.849867 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:42.849801 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:42.850369 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:42.850342 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:03:52.849857 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:52.849795 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:03:52.850406 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:03:52.850284 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:04:02.850193 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:02.850146 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:04:02.850748 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:02.850606 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:04:12.849986 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:12.849947 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:04:12.850543 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:12.850517 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:04:22.850461 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:22.850421 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:04:22.851065 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:22.851040 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:04:32.850088 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:32.850054 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:04:32.850584 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:32.850378 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:04:44.873745 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:44.873708 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d_9cae34c1-3cb9-452b-9ec4-3cbd4077479a/kserve-container/0.log" May 20 14:04:45.050870 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.050812 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:04:45.051425 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.051391 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" containerID="cri-o://9b0eb757ca78aff3220435a4af0eeba08729dc46702762cc5280f5ec7e60cfb3" gracePeriod=30 May 20 14:04:45.051550 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.051418 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" containerID="cri-o://b05f01e8acb85ffae5fea5284ad2f4e6ebff036e57d36dc9e25dd1c695f64ab1" gracePeriod=30 May 20 14:04:45.051550 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.051443 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" containerID="cri-o://ae5ac74c01dab31506a768a6db49fb6f2d00466fb6300ec0c613e22af56190c8" gracePeriod=30 May 20 14:04:45.095825 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.095786 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:04:45.099742 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.099717 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.102103 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.102083 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-3983e-predictor-serving-cert\"" May 20 14:04:45.102182 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.102088 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\"" May 20 14:04:45.107795 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.107772 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:04:45.136244 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.136170 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:04:45.136513 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.136483 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kserve-container" containerID="cri-o://d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" gracePeriod=30 May 20 14:04:45.136619 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.136489 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kube-rbac-proxy" containerID="cri-o://1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" gracePeriod=30 May 20 14:04:45.143390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.143365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.143480 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.143400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.143480 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.143435 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2mh\" (UniqueName: \"kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.143549 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.143480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.143609 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.143589 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244220 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244126 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244220 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244226 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244220 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244543 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244309 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244543 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244346 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2mh\" (UniqueName: \"kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244663 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244621 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.244899 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.244875 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.245296 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.245272 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.247014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.246990 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.251771 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.251745 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2mh\" (UniqueName: \"kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh\") pod \"isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.372408 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.372384 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:04:45.412552 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.412486 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:45.446495 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.446460 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " May 20 14:04:45.446655 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.446542 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") pod \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " May 20 14:04:45.446655 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.446612 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hknv2\" (UniqueName: \"kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2\") pod \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\" (UID: \"9cae34c1-3cb9-452b-9ec4-3cbd4077479a\") " May 20 14:04:45.446901 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.446871 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-cbb25-kube-rbac-proxy-sar-config") pod "9cae34c1-3cb9-452b-9ec4-3cbd4077479a" (UID: "9cae34c1-3cb9-452b-9ec4-3cbd4077479a"). InnerVolumeSpecName "message-dumper-raw-cbb25-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:04:45.448799 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.448753 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2" (OuterVolumeSpecName: "kube-api-access-hknv2") pod "9cae34c1-3cb9-452b-9ec4-3cbd4077479a" (UID: "9cae34c1-3cb9-452b-9ec4-3cbd4077479a"). InnerVolumeSpecName "kube-api-access-hknv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:04:45.449055 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.449020 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9cae34c1-3cb9-452b-9ec4-3cbd4077479a" (UID: "9cae34c1-3cb9-452b-9ec4-3cbd4077479a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:04:45.540856 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.540777 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:04:45.543177 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:04:45.543154 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8b523f_a553_40ea_9307_7295088932f0.slice/crio-0da974eb37b5510c2f2e75a7e8f1da22a7c2cdae7b596e840eef85aba89836e3 WatchSource:0}: Error finding container 0da974eb37b5510c2f2e75a7e8f1da22a7c2cdae7b596e840eef85aba89836e3: Status 404 returned error can't find the container with id 0da974eb37b5510c2f2e75a7e8f1da22a7c2cdae7b596e840eef85aba89836e3 May 20 14:04:45.548170 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.548150 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknv2\" (UniqueName: \"kubernetes.io/projected/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-kube-api-access-hknv2\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:04:45.548230 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.548174 2554 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-message-dumper-raw-cbb25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:04:45.548230 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:45.548187 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cae34c1-3cb9-452b-9ec4-3cbd4077479a-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:04:46.094814 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.094782 2554 generic.go:358] "Generic (PLEG): container finished" podID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerID="ae5ac74c01dab31506a768a6db49fb6f2d00466fb6300ec0c613e22af56190c8" exitCode=2 May 20 14:04:46.095257 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.094870 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerDied","Data":"ae5ac74c01dab31506a768a6db49fb6f2d00466fb6300ec0c613e22af56190c8"} May 20 14:04:46.096391 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096367 2554 generic.go:358] "Generic (PLEG): container finished" podID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerID="1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" exitCode=2 May 20 14:04:46.096391 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096388 2554 generic.go:358] "Generic (PLEG): container finished" podID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerID="d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" exitCode=2 May 20 14:04:46.096554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096447 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerDied","Data":"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91"} May 20 14:04:46.096554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096463 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" May 20 14:04:46.096554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096481 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerDied","Data":"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4"} May 20 14:04:46.096554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096498 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d" event={"ID":"9cae34c1-3cb9-452b-9ec4-3cbd4077479a","Type":"ContainerDied","Data":"a2531e57578c6fbb3deb69d910e24bcb44a337296168c78e2cf9065dd8f252e6"} May 20 14:04:46.096554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.096516 2554 scope.go:117] "RemoveContainer" containerID="1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" May 20 14:04:46.098114 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.098076 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerStarted","Data":"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624"} May 20 14:04:46.098114 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.098109 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerStarted","Data":"0da974eb37b5510c2f2e75a7e8f1da22a7c2cdae7b596e840eef85aba89836e3"} May 20 14:04:46.105488 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.105462 2554 scope.go:117] "RemoveContainer" containerID="d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" May 20 14:04:46.115374 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.115349 2554 scope.go:117] "RemoveContainer" containerID="1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" May 20 14:04:46.115694 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:04:46.115672 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91\": container with ID starting with 1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91 not found: ID does not exist" containerID="1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" May 20 14:04:46.115779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.115703 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91"} err="failed to get container status \"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91\": rpc error: code = NotFound desc = could not find container \"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91\": container with ID starting with 1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91 not found: ID does not exist" May 20 14:04:46.115779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.115735 2554 scope.go:117] "RemoveContainer" containerID="d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" May 20 14:04:46.116049 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:04:46.116029 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4\": container with ID starting with d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4 not found: ID does not exist" containerID="d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" May 20 14:04:46.116104 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.116061 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4"} err="failed to get container status \"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4\": rpc error: code = NotFound desc = could not find container \"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4\": container with ID starting with d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4 not found: ID does not exist" May 20 14:04:46.116104 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.116081 2554 scope.go:117] "RemoveContainer" containerID="1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91" May 20 14:04:46.116350 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.116330 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91"} err="failed to get container status \"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91\": rpc error: code = NotFound desc = could not find container \"1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91\": container with ID starting with 1d4431e9b7d95e674180f38a803a7cdea9bc6a23e71c14b21d299a674f451e91 not found: ID does not exist" May 20 14:04:46.116424 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.116350 2554 scope.go:117] "RemoveContainer" containerID="d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4" May 20 14:04:46.116585 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.116561 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4"} err="failed to get container status \"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4\": rpc error: code = NotFound desc = could not find container \"d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4\": container with ID starting with d5f5cd2dc8c6a234456472e5d27d0d75355ab1dffb5b968ef0bbbd582754eca4 not found: ID does not exist" May 20 14:04:46.130824 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.130797 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:04:46.133942 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.133918 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cbb25-predictor-74678f7ffd-spr2d"] May 20 14:04:46.927577 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:46.927542 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" path="/var/lib/kubelet/pods/9cae34c1-3cb9-452b-9ec4-3cbd4077479a/volumes" May 20 14:04:47.845813 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:47.845773 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:04:50.116548 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:50.116513 2554 generic.go:358] "Generic (PLEG): container finished" podID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerID="9b0eb757ca78aff3220435a4af0eeba08729dc46702762cc5280f5ec7e60cfb3" exitCode=0 May 20 14:04:50.116948 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:50.116586 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerDied","Data":"9b0eb757ca78aff3220435a4af0eeba08729dc46702762cc5280f5ec7e60cfb3"} May 20 14:04:50.117963 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:50.117939 2554 generic.go:358] "Generic (PLEG): container finished" podID="cd8b523f-a553-40ea-9307-7295088932f0" containerID="c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624" exitCode=0 May 20 14:04:50.118082 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:50.117993 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerDied","Data":"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624"} May 20 14:04:51.122942 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:51.122909 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerStarted","Data":"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92"} May 20 14:04:51.123317 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:51.122949 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerStarted","Data":"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5"} May 20 14:04:51.123317 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:51.123161 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:51.143877 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:51.143813 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podStartSLOduration=6.143796826 podStartE2EDuration="6.143796826s" podCreationTimestamp="2026-05-20 14:04:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:04:51.14168027 +0000 UTC m=+886.848112331" watchObservedRunningTime="2026-05-20 14:04:51.143796826 +0000 UTC m=+886.850228887" May 20 14:04:52.126674 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:52.126594 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:52.128050 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:52.128022 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:04:52.845759 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:52.845710 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:04:52.850052 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:52.850026 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:04:52.850433 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:52.850412 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:04:53.130208 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:53.130126 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:04:57.845772 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:57.845731 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:04:57.846169 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:57.845895 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:04:58.134569 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:58.134487 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:04:58.135144 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:04:58.135114 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:02.845648 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:02.845606 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:05:02.849939 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:02.849912 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:05:02.850345 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:02.850320 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:05:04.881828 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:04.881797 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:05:04.882485 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:04.882456 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:05:07.845004 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:07.844956 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:05:08.135330 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:08.135240 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:12.845966 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:12.845922 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" May 20 14:05:12.850342 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:12.850309 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" May 20 14:05:12.850494 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:12.850460 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:05:12.850707 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:12.850681 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 20 14:05:12.850788 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:12.850769 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:05:15.204917 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.204885 2554 generic.go:358] "Generic (PLEG): container finished" podID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerID="b05f01e8acb85ffae5fea5284ad2f4e6ebff036e57d36dc9e25dd1c695f64ab1" exitCode=0 May 20 14:05:15.205295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.204947 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerDied","Data":"b05f01e8acb85ffae5fea5284ad2f4e6ebff036e57d36dc9e25dd1c695f64ab1"} May 20 14:05:15.205295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.204980 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" event={"ID":"e2977e91-a2b4-4808-8558-4184e458a9b2","Type":"ContainerDied","Data":"615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db"} May 20 14:05:15.205295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.204990 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="615ae7c66eef061e6e90abf77e57883fb7b29e80ade35d8896adef7082d5e8db" May 20 14:05:15.205989 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.205971 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:05:15.299949 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.299916 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert\") pod \"e2977e91-a2b4-4808-8558-4184e458a9b2\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " May 20 14:05:15.300138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.299973 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") pod \"e2977e91-a2b4-4808-8558-4184e458a9b2\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " May 20 14:05:15.300138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300051 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26bwf\" (UniqueName: \"kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf\") pod \"e2977e91-a2b4-4808-8558-4184e458a9b2\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " May 20 14:05:15.300138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300085 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location\") pod \"e2977e91-a2b4-4808-8558-4184e458a9b2\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " May 20 14:05:15.300138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300133 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\") pod \"e2977e91-a2b4-4808-8558-4184e458a9b2\" (UID: \"e2977e91-a2b4-4808-8558-4184e458a9b2\") " May 20 14:05:15.300356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300336 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2977e91-a2b4-4808-8558-4184e458a9b2" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:05:15.300433 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300410 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2977e91-a2b4-4808-8558-4184e458a9b2" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:05:15.300491 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300436 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:05:15.300545 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.300494 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config") pod "e2977e91-a2b4-4808-8558-4184e458a9b2" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2"). InnerVolumeSpecName "isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:05:15.302313 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.302291 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2977e91-a2b4-4808-8558-4184e458a9b2" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:05:15.302670 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.302656 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf" (OuterVolumeSpecName: "kube-api-access-26bwf") pod "e2977e91-a2b4-4808-8558-4184e458a9b2" (UID: "e2977e91-a2b4-4808-8558-4184e458a9b2"). InnerVolumeSpecName "kube-api-access-26bwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:05:15.401645 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.401617 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26bwf\" (UniqueName: \"kubernetes.io/projected/e2977e91-a2b4-4808-8558-4184e458a9b2-kube-api-access-26bwf\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:05:15.401645 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.401642 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2977e91-a2b4-4808-8558-4184e458a9b2-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:05:15.401828 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.401653 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2977e91-a2b4-4808-8558-4184e458a9b2-isvc-logger-raw-cbb25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:05:15.401828 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:15.401664 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2977e91-a2b4-4808-8558-4184e458a9b2-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:05:16.208338 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:16.208307 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp" May 20 14:05:16.228036 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:16.228007 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:05:16.233088 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:16.233062 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cbb25-predictor-5959584f64-9zrpp"] May 20 14:05:16.927176 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:16.927136 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" path="/var/lib/kubelet/pods/e2977e91-a2b4-4808-8558-4184e458a9b2/volumes" May 20 14:05:18.135518 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:18.135482 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:28.135734 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:28.135693 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:38.135958 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:38.135916 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:48.135539 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:48.135499 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:05:58.135852 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:05:58.135793 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:08.135277 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:08.135236 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:18.135638 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:18.135588 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:20.923104 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:20.923043 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:30.923628 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:30.923531 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:40.923156 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:40.923105 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:06:50.923045 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:06:50.922991 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:07:00.923068 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:00.923005 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:07:10.927305 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:10.927274 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:07:15.242847 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.242801 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:07:15.243275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.243227 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" containerID="cri-o://7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5" gracePeriod=30 May 20 14:07:15.243340 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.243275 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" containerID="cri-o://2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92" gracePeriod=30 May 20 14:07:15.345500 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345464 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:07:15.345893 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345874 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" May 20 14:07:15.345893 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345891 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345908 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345914 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345923 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kube-rbac-proxy" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345930 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kube-rbac-proxy" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345949 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="storage-initializer" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345954 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="storage-initializer" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345961 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kserve-container" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345966 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kserve-container" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345979 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.345984 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" May 20 14:07:15.346043 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.346046 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="agent" May 20 14:07:15.346366 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.346055 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kube-rbac-proxy" May 20 14:07:15.346366 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.346063 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kserve-container" May 20 14:07:15.346366 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.346070 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="9cae34c1-3cb9-452b-9ec4-3cbd4077479a" containerName="kserve-container" May 20 14:07:15.346366 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.346076 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2977e91-a2b4-4808-8558-4184e458a9b2" containerName="kube-rbac-proxy" May 20 14:07:15.349377 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.349353 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.351784 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.351760 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cc9515-predictor-serving-cert\"" May 20 14:07:15.352194 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.352167 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cc9515-kube-rbac-proxy-sar-config\"" May 20 14:07:15.360386 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.360356 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:07:15.444248 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.444203 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5fr\" (UniqueName: \"kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.444464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.444269 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.444464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.444331 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.444464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.444361 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.444464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.444426 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545260 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545157 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545265 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545298 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545360 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545390 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5fr\" (UniqueName: \"kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.545648 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:07:15.545485 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-cc9515-predictor-serving-cert: secret "isvc-primary-cc9515-predictor-serving-cert" not found May 20 14:07:15.545648 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:07:15.545577 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls podName:57e8517c-0f31-46df-a82f-354bff9abb61 nodeName:}" failed. No retries permitted until 2026-05-20 14:07:16.045552497 +0000 UTC m=+1031.751984552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls") pod "isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61") : secret "isvc-primary-cc9515-predictor-serving-cert" not found May 20 14:07:15.545648 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545610 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.546019 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.545999 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.546070 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.546040 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.556606 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.556580 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5fr\" (UniqueName: \"kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:15.598107 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.598072 2554 generic.go:358] "Generic (PLEG): container finished" podID="cd8b523f-a553-40ea-9307-7295088932f0" containerID="2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92" exitCode=2 May 20 14:07:15.598290 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:15.598156 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerDied","Data":"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92"} May 20 14:07:16.049686 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.049643 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:16.052294 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.052268 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") pod \"isvc-primary-cc9515-predictor-79d47ccd86-dm7kl\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:16.262330 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.262289 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:16.396404 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.396372 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:07:16.399209 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:07:16.399180 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e8517c_0f31_46df_a82f_354bff9abb61.slice/crio-0708eb1b81f41ace739a1fc81a9cca6dec6ba6dd3d1536ec501729ba539906dc WatchSource:0}: Error finding container 0708eb1b81f41ace739a1fc81a9cca6dec6ba6dd3d1536ec501729ba539906dc: Status 404 returned error can't find the container with id 0708eb1b81f41ace739a1fc81a9cca6dec6ba6dd3d1536ec501729ba539906dc May 20 14:07:16.604431 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.604306 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerStarted","Data":"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464"} May 20 14:07:16.604431 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:16.604367 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerStarted","Data":"0708eb1b81f41ace739a1fc81a9cca6dec6ba6dd3d1536ec501729ba539906dc"} May 20 14:07:18.130458 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:18.130409 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" May 20 14:07:20.618628 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:20.618593 2554 generic.go:358] "Generic (PLEG): container finished" podID="57e8517c-0f31-46df-a82f-354bff9abb61" containerID="d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464" exitCode=0 May 20 14:07:20.619050 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:20.618675 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerDied","Data":"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464"} May 20 14:07:20.923130 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:20.923030 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" May 20 14:07:21.624549 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:21.624511 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerStarted","Data":"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec"} May 20 14:07:21.624549 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:21.624551 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerStarted","Data":"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4"} May 20 14:07:21.625030 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:21.624860 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:21.654661 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:21.649631 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podStartSLOduration=6.649598857 podStartE2EDuration="6.649598857s" podCreationTimestamp="2026-05-20 14:07:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:07:21.645973135 +0000 UTC m=+1037.352405196" watchObservedRunningTime="2026-05-20 14:07:21.649598857 +0000 UTC m=+1037.356030920" May 20 14:07:22.628108 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:22.628073 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:22.629338 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:22.629306 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:07:23.130818 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:23.130774 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" May 20 14:07:23.630867 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:23.630813 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:07:25.382175 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.382153 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:07:25.431039 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431002 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\") pod \"cd8b523f-a553-40ea-9307-7295088932f0\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " May 20 14:07:25.431235 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431070 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm2mh\" (UniqueName: \"kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh\") pod \"cd8b523f-a553-40ea-9307-7295088932f0\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " May 20 14:07:25.431235 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431181 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls\") pod \"cd8b523f-a553-40ea-9307-7295088932f0\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " May 20 14:07:25.431356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431244 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert\") pod \"cd8b523f-a553-40ea-9307-7295088932f0\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " May 20 14:07:25.431356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431291 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location\") pod \"cd8b523f-a553-40ea-9307-7295088932f0\" (UID: \"cd8b523f-a553-40ea-9307-7295088932f0\") " May 20 14:07:25.431542 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431430 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config") pod "cd8b523f-a553-40ea-9307-7295088932f0" (UID: "cd8b523f-a553-40ea-9307-7295088932f0"). InnerVolumeSpecName "isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:07:25.431685 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431596 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd8b523f-a553-40ea-9307-7295088932f0" (UID: "cd8b523f-a553-40ea-9307-7295088932f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:07:25.431685 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.431617 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd8b523f-a553-40ea-9307-7295088932f0" (UID: "cd8b523f-a553-40ea-9307-7295088932f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:07:25.433308 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.433284 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd8b523f-a553-40ea-9307-7295088932f0" (UID: "cd8b523f-a553-40ea-9307-7295088932f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:07:25.433400 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.433331 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh" (OuterVolumeSpecName: "kube-api-access-rm2mh") pod "cd8b523f-a553-40ea-9307-7295088932f0" (UID: "cd8b523f-a553-40ea-9307-7295088932f0"). InnerVolumeSpecName "kube-api-access-rm2mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:07:25.532713 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.532614 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:07:25.532713 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.532650 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd8b523f-a553-40ea-9307-7295088932f0-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:07:25.532713 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.532666 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd8b523f-a553-40ea-9307-7295088932f0-isvc-sklearn-scale-raw-3983e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:07:25.532713 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.532675 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2mh\" (UniqueName: \"kubernetes.io/projected/cd8b523f-a553-40ea-9307-7295088932f0-kube-api-access-rm2mh\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:07:25.532713 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.532686 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8b523f-a553-40ea-9307-7295088932f0-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:07:25.642101 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.642063 2554 generic.go:358] "Generic (PLEG): container finished" podID="cd8b523f-a553-40ea-9307-7295088932f0" containerID="7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5" exitCode=0 May 20 14:07:25.642269 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.642150 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" May 20 14:07:25.642269 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.642154 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerDied","Data":"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5"} May 20 14:07:25.642269 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.642206 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62" event={"ID":"cd8b523f-a553-40ea-9307-7295088932f0","Type":"ContainerDied","Data":"0da974eb37b5510c2f2e75a7e8f1da22a7c2cdae7b596e840eef85aba89836e3"} May 20 14:07:25.642269 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.642231 2554 scope.go:117] "RemoveContainer" containerID="2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92" May 20 14:07:25.650991 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.650971 2554 scope.go:117] "RemoveContainer" containerID="7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5" May 20 14:07:25.658773 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.658753 2554 scope.go:117] "RemoveContainer" containerID="c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624" May 20 14:07:25.663700 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.663670 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:07:25.667266 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.667239 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-3983e-predictor-6d6c6bcdd5-2cj62"] May 20 14:07:25.667462 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.667445 2554 scope.go:117] "RemoveContainer" containerID="2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92" May 20 14:07:25.667777 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:07:25.667750 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92\": container with ID starting with 2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92 not found: ID does not exist" containerID="2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92" May 20 14:07:25.667823 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.667789 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92"} err="failed to get container status \"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92\": rpc error: code = NotFound desc = could not find container \"2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92\": container with ID starting with 2c8e9974601130ccd061ee035e5239760aa5559f90548413f14b769b0688fb92 not found: ID does not exist" May 20 14:07:25.667823 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.667808 2554 scope.go:117] "RemoveContainer" containerID="7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5" May 20 14:07:25.668109 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:07:25.668082 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5\": container with ID starting with 7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5 not found: ID does not exist" containerID="7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5" May 20 14:07:25.668162 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.668114 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5"} err="failed to get container status \"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5\": rpc error: code = NotFound desc = could not find container \"7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5\": container with ID starting with 7a7daec177f00302e5f25169682872cbf8d58b37272952614876c490b6ac4cf5 not found: ID does not exist" May 20 14:07:25.668162 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.668132 2554 scope.go:117] "RemoveContainer" containerID="c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624" May 20 14:07:25.668396 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:07:25.668378 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624\": container with ID starting with c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624 not found: ID does not exist" containerID="c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624" May 20 14:07:25.668483 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:25.668403 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624"} err="failed to get container status \"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624\": rpc error: code = NotFound desc = could not find container \"c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624\": container with ID starting with c8dab2f6cc0bad251fbbe473daa21cb689e3c359c61e2163588fe2202f8ea624 not found: ID does not exist" May 20 14:07:26.932618 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:26.932582 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8b523f-a553-40ea-9307-7295088932f0" path="/var/lib/kubelet/pods/cd8b523f-a553-40ea-9307-7295088932f0/volumes" May 20 14:07:28.636030 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:28.635998 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:07:28.636641 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:28.636615 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:07:38.636781 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:38.636740 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:07:48.636511 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:48.636472 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:07:58.636809 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:07:58.636718 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:08:08.636981 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:08.636933 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:08:18.636734 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:18.636696 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" May 20 14:08:28.637688 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:28.637653 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:08:35.470591 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470558 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470947 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470959 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470968 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470973 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470982 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="storage-initializer" May 20 14:08:35.471014 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.470988 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="storage-initializer" May 20 14:08:35.471203 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.471057 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kserve-container" May 20 14:08:35.471203 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.471069 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd8b523f-a553-40ea-9307-7295088932f0" containerName="kube-rbac-proxy" May 20 14:08:35.474135 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.474118 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.476419 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.476388 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cc9515-predictor-serving-cert\"" May 20 14:08:35.476419 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.476408 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cc9515\"" May 20 14:08:35.476595 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.476482 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cc9515-dockercfg-hv9jg\"" May 20 14:08:35.476676 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.476663 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\"" May 20 14:08:35.486423 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.486399 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:35.531164 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.531129 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwjk\" (UniqueName: \"kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.531356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.531195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.531356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.531219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.531356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.531252 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.531356 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.531290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632042 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632004 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwjk\" (UniqueName: \"kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632234 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632061 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632234 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632079 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632234 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632101 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632234 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632126 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632511 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632491 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632759 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632738 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.632807 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.632768 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.634889 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.634868 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.640418 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.640394 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwjk\" (UniqueName: \"kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk\") pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.784554 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.784459 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:35.915501 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.915478 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:35.917674 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:08:35.917644 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e3ed0e_95e2_4dbc_8968_2abe3a782dae.slice/crio-e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d WatchSource:0}: Error finding container e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d: Status 404 returned error can't find the container with id e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d May 20 14:08:35.919528 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:35.919513 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 20 14:08:36.864346 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:36.864306 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerStarted","Data":"ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc"} May 20 14:08:36.864346 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:36.864347 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerStarted","Data":"e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d"} May 20 14:08:38.872730 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:38.872703 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/0.log" May 20 14:08:38.873078 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:38.872743 2554 generic.go:358] "Generic (PLEG): container finished" podID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerID="ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc" exitCode=1 May 20 14:08:38.873078 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:38.872817 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerDied","Data":"ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc"} May 20 14:08:39.883236 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:39.883208 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/0.log" May 20 14:08:39.883647 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:39.883330 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerStarted","Data":"e6a73cce03699fb53a843077fd747b026bdc2109253d1ac430038a78ae03bddf"} May 20 14:08:45.903603 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.903530 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/1.log" May 20 14:08:45.903987 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.903907 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/0.log" May 20 14:08:45.903987 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.903937 2554 generic.go:358] "Generic (PLEG): container finished" podID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerID="e6a73cce03699fb53a843077fd747b026bdc2109253d1ac430038a78ae03bddf" exitCode=1 May 20 14:08:45.904068 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.903993 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerDied","Data":"e6a73cce03699fb53a843077fd747b026bdc2109253d1ac430038a78ae03bddf"} May 20 14:08:45.904068 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.904026 2554 scope.go:117] "RemoveContainer" containerID="ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc" May 20 14:08:45.904449 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:45.904429 2554 scope.go:117] "RemoveContainer" containerID="ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc" May 20 14:08:45.914677 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:45.914642 2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_kserve-ci-e2e-test_70e3ed0e-95e2-4dbc-8968-2abe3a782dae_0 in pod sandbox e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d from index: no such id: 'ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc'" containerID="ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc" May 20 14:08:45.914781 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:45.914696 2554 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_kserve-ci-e2e-test_70e3ed0e-95e2-4dbc-8968-2abe3a782dae_0 in pod sandbox e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d from index: no such id: 'ac256e84ab4bb97b6a8ab8f482c6385d0d5809bff32759a5fd000b34bb5641fc'; Skipping pod \"isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_kserve-ci-e2e-test(70e3ed0e-95e2-4dbc-8968-2abe3a782dae)\"" logger="UnhandledError" May 20 14:08:45.916051 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:45.916032 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_kserve-ci-e2e-test(70e3ed0e-95e2-4dbc-8968-2abe3a782dae)\"" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" May 20 14:08:46.908616 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:46.908590 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/1.log" May 20 14:08:49.518858 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.518799 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:49.567574 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.567542 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:08:49.567902 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.567878 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" containerID="cri-o://7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4" gracePeriod=30 May 20 14:08:49.568001 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.567927 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kube-rbac-proxy" containerID="cri-o://6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec" gracePeriod=30 May 20 14:08:49.666473 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.666436 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:08:49.671033 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.671014 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.673412 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.673391 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\"" May 20 14:08:49.673513 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.673429 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-bb9d97\"" May 20 14:08:49.673513 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.673451 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-bb9d97-predictor-serving-cert\"" May 20 14:08:49.673513 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.673461 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-bb9d97-dockercfg-r2ssp\"" May 20 14:08:49.680119 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.680096 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:08:49.694483 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.694464 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/1.log" May 20 14:08:49.694574 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.694531 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:49.858872 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.858733 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert\") pod \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " May 20 14:08:49.858872 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.858798 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls\") pod \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " May 20 14:08:49.858872 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.858825 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location\") pod \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " May 20 14:08:49.859171 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.858905 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config\") pod \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " May 20 14:08:49.859171 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.858950 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldwjk\" (UniqueName: \"kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk\") pod \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\" (UID: \"70e3ed0e-95e2-4dbc-8968-2abe3a782dae\") " May 20 14:08:49.859171 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.859171 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859155 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859178 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70e3ed0e-95e2-4dbc-8968-2abe3a782dae" (UID: "70e3ed0e-95e2-4dbc-8968-2abe3a782dae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859204 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4567h\" (UniqueName: \"kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859281 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70e3ed0e-95e2-4dbc-8968-2abe3a782dae" (UID: "70e3ed0e-95e2-4dbc-8968-2abe3a782dae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859306 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.859373 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859324 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-cc9515-kube-rbac-proxy-sar-config") pod "70e3ed0e-95e2-4dbc-8968-2abe3a782dae" (UID: "70e3ed0e-95e2-4dbc-8968-2abe3a782dae"). InnerVolumeSpecName "isvc-secondary-cc9515-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:08:49.859636 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859411 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:49.859636 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859426 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:49.859636 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.859436 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-isvc-secondary-cc9515-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:49.861243 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.861222 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk" (OuterVolumeSpecName: "kube-api-access-ldwjk") pod "70e3ed0e-95e2-4dbc-8968-2abe3a782dae" (UID: "70e3ed0e-95e2-4dbc-8968-2abe3a782dae"). InnerVolumeSpecName "kube-api-access-ldwjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:08:49.861306 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.861282 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70e3ed0e-95e2-4dbc-8968-2abe3a782dae" (UID: "70e3ed0e-95e2-4dbc-8968-2abe3a782dae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:08:49.919029 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.919001 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg_70e3ed0e-95e2-4dbc-8968-2abe3a782dae/storage-initializer/1.log" May 20 14:08:49.919206 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.919084 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" event={"ID":"70e3ed0e-95e2-4dbc-8968-2abe3a782dae","Type":"ContainerDied","Data":"e0b3db1cbc084823df7b13f6f794cefd896d68f5ffe4b3f113449997e46ec70d"} May 20 14:08:49.919206 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.919115 2554 scope.go:117] "RemoveContainer" containerID="e6a73cce03699fb53a843077fd747b026bdc2109253d1ac430038a78ae03bddf" May 20 14:08:49.919206 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.919115 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg" May 20 14:08:49.921311 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.921285 2554 generic.go:358] "Generic (PLEG): container finished" podID="57e8517c-0f31-46df-a82f-354bff9abb61" containerID="6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec" exitCode=2 May 20 14:08:49.921427 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.921358 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerDied","Data":"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec"} May 20 14:08:49.957620 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.957589 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.959883 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.959924 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.959958 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4567h\" (UniqueName: \"kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960017 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960091 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960153 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960151 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:49.960610 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960167 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwjk\" (UniqueName: \"kubernetes.io/projected/70e3ed0e-95e2-4dbc-8968-2abe3a782dae-kube-api-access-ldwjk\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:49.960610 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:49.960187 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-serving-cert: secret "isvc-init-fail-bb9d97-predictor-serving-cert" not found May 20 14:08:49.960610 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:49.960269 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls podName:75adfcb7-d701-40c1-8454-d83299a69a8e nodeName:}" failed. No retries permitted until 2026-05-20 14:08:50.460246263 +0000 UTC m=+1126.166678308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls") pod "isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e") : secret "isvc-init-fail-bb9d97-predictor-serving-cert" not found May 20 14:08:49.960808 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960780 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.960946 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960901 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.961020 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.960972 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:49.961734 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.961713 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cc9515-predictor-6bfcd87c77-h76gg"] May 20 14:08:49.972686 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:49.972666 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4567h\" (UniqueName: \"kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:50.463997 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.463957 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:50.466465 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.466443 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") pod \"isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:50.593006 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.592974 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:08:50.725027 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.724952 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:08:50.727946 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:08:50.727921 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75adfcb7_d701_40c1_8454_d83299a69a8e.slice/crio-c2aebeb8967f3cda08cdd6e1541bfc77ad2c24960822dca594868c45744abf86 WatchSource:0}: Error finding container c2aebeb8967f3cda08cdd6e1541bfc77ad2c24960822dca594868c45744abf86: Status 404 returned error can't find the container with id c2aebeb8967f3cda08cdd6e1541bfc77ad2c24960822dca594868c45744abf86 May 20 14:08:50.926670 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.926631 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" path="/var/lib/kubelet/pods/70e3ed0e-95e2-4dbc-8968-2abe3a782dae/volumes" May 20 14:08:50.927011 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.926989 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerStarted","Data":"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779"} May 20 14:08:50.927075 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:50.927019 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerStarted","Data":"c2aebeb8967f3cda08cdd6e1541bfc77ad2c24960822dca594868c45744abf86"} May 20 14:08:53.631223 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:53.631177 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused" May 20 14:08:54.109479 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.109458 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:08:54.198789 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.198759 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw5fr\" (UniqueName: \"kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr\") pod \"57e8517c-0f31-46df-a82f-354bff9abb61\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " May 20 14:08:54.198978 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.198868 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location\") pod \"57e8517c-0f31-46df-a82f-354bff9abb61\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " May 20 14:08:54.198978 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.198894 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert\") pod \"57e8517c-0f31-46df-a82f-354bff9abb61\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " May 20 14:08:54.198978 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.198942 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") pod \"57e8517c-0f31-46df-a82f-354bff9abb61\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " May 20 14:08:54.198978 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.198971 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config\") pod \"57e8517c-0f31-46df-a82f-354bff9abb61\" (UID: \"57e8517c-0f31-46df-a82f-354bff9abb61\") " May 20 14:08:54.199293 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.199260 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57e8517c-0f31-46df-a82f-354bff9abb61" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:08:54.199397 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.199303 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57e8517c-0f31-46df-a82f-354bff9abb61" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:08:54.199397 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.199338 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-cc9515-kube-rbac-proxy-sar-config") pod "57e8517c-0f31-46df-a82f-354bff9abb61" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61"). InnerVolumeSpecName "isvc-primary-cc9515-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:08:54.201107 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.201087 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57e8517c-0f31-46df-a82f-354bff9abb61" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:08:54.201193 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.201105 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr" (OuterVolumeSpecName: "kube-api-access-kw5fr") pod "57e8517c-0f31-46df-a82f-354bff9abb61" (UID: "57e8517c-0f31-46df-a82f-354bff9abb61"). InnerVolumeSpecName "kube-api-access-kw5fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:08:54.299505 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.299474 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e8517c-0f31-46df-a82f-354bff9abb61-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:54.299505 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.299501 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:54.299676 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.299518 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e8517c-0f31-46df-a82f-354bff9abb61-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:54.299676 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.299533 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-cc9515-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e8517c-0f31-46df-a82f-354bff9abb61-isvc-primary-cc9515-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:54.299676 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.299547 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw5fr\" (UniqueName: \"kubernetes.io/projected/57e8517c-0f31-46df-a82f-354bff9abb61-kube-api-access-kw5fr\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:08:54.942158 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.942121 2554 generic.go:358] "Generic (PLEG): container finished" podID="57e8517c-0f31-46df-a82f-354bff9abb61" containerID="7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4" exitCode=0 May 20 14:08:54.942502 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.942175 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerDied","Data":"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4"} May 20 14:08:54.942502 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.942202 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" May 20 14:08:54.942502 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.942213 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl" event={"ID":"57e8517c-0f31-46df-a82f-354bff9abb61","Type":"ContainerDied","Data":"0708eb1b81f41ace739a1fc81a9cca6dec6ba6dd3d1536ec501729ba539906dc"} May 20 14:08:54.942502 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.942228 2554 scope.go:117] "RemoveContainer" containerID="6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec" May 20 14:08:54.950361 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.950343 2554 scope.go:117] "RemoveContainer" containerID="7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4" May 20 14:08:54.957534 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.957517 2554 scope.go:117] "RemoveContainer" containerID="d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464" May 20 14:08:54.961934 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.961915 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:08:54.966470 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.966452 2554 scope.go:117] "RemoveContainer" containerID="6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec" May 20 14:08:54.966736 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:54.966716 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec\": container with ID starting with 6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec not found: ID does not exist" containerID="6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec" May 20 14:08:54.966791 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.966745 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec"} err="failed to get container status \"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec\": rpc error: code = NotFound desc = could not find container \"6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec\": container with ID starting with 6d642993660ac000df74fec3a7fc28b3b3e032229ce84aa1086407665c5315ec not found: ID does not exist" May 20 14:08:54.966791 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.966762 2554 scope.go:117] "RemoveContainer" containerID="7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4" May 20 14:08:54.967007 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.966986 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cc9515-predictor-79d47ccd86-dm7kl"] May 20 14:08:54.967047 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:54.967006 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4\": container with ID starting with 7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4 not found: ID does not exist" containerID="7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4" May 20 14:08:54.967047 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.967028 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4"} err="failed to get container status \"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4\": rpc error: code = NotFound desc = could not find container \"7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4\": container with ID starting with 7254699e9b62d801e63097f70218096b79ec765c6eae994ad72bf189bbc585c4 not found: ID does not exist" May 20 14:08:54.967047 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.967040 2554 scope.go:117] "RemoveContainer" containerID="d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464" May 20 14:08:54.967245 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:54.967232 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464\": container with ID starting with d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464 not found: ID does not exist" containerID="d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464" May 20 14:08:54.967283 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:54.967249 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464"} err="failed to get container status \"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464\": rpc error: code = NotFound desc = could not find container \"d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464\": container with ID starting with d8de9010fc1716411fcd554b18d48ff18547fbd08221600d12c63a32430b9464 not found: ID does not exist" May 20 14:08:55.948000 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:55.947965 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/0.log" May 20 14:08:55.948000 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:55.948002 2554 generic.go:358] "Generic (PLEG): container finished" podID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerID="2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779" exitCode=1 May 20 14:08:55.948436 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:55.948090 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerDied","Data":"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779"} May 20 14:08:56.926887 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:56.926829 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" path="/var/lib/kubelet/pods/57e8517c-0f31-46df-a82f-354bff9abb61/volumes" May 20 14:08:56.953074 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:56.953038 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/0.log" May 20 14:08:56.953436 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:56.953162 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerStarted","Data":"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521"} May 20 14:08:59.653255 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.653216 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:08:59.653789 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.653492 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" containerID="cri-o://bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521" gracePeriod=30 May 20 14:08:59.778902 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.778816 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:08:59.779231 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779219 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779233 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779252 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="storage-initializer" May 20 14:08:59.779275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779258 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="storage-initializer" May 20 14:08:59.779275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779270 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kube-rbac-proxy" May 20 14:08:59.779275 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779275 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kube-rbac-proxy" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779282 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779287 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779300 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779305 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779358 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779389 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="70e3ed0e-95e2-4dbc-8968-2abe3a782dae" containerName="storage-initializer" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779397 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kserve-container" May 20 14:08:59.779526 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.779407 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="57e8517c-0f31-46df-a82f-354bff9abb61" containerName="kube-rbac-proxy" May 20 14:08:59.782512 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.782496 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.785140 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.785113 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1b556-predictor-serving-cert\"" May 20 14:08:59.785282 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.785184 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" May 20 14:08:59.785282 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.785212 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1b556-kube-rbac-proxy-sar-config\"" May 20 14:08:59.785415 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.785293 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-25znm\"" May 20 14:08:59.793062 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.793040 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:08:59.849090 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.849046 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.849295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.849106 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.849295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.849138 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.849295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.849228 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-1b556-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.849295 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.849284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbqz\" (UniqueName: \"kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949653 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.949621 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbqz\" (UniqueName: \"kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949882 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.949664 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949882 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.949694 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949882 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.949715 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949882 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.949755 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-1b556-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.949882 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:59.949793 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-1b556-predictor-serving-cert: secret "raw-sklearn-1b556-predictor-serving-cert" not found May 20 14:08:59.950160 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:08:59.949898 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls podName:b4bf8885-c618-4380-bbde-a36691630392 nodeName:}" failed. No retries permitted until 2026-05-20 14:09:00.449873023 +0000 UTC m=+1136.156305063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls") pod "raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" (UID: "b4bf8885-c618-4380-bbde-a36691630392") : secret "raw-sklearn-1b556-predictor-serving-cert" not found May 20 14:08:59.950160 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.950129 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.950385 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.950366 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-1b556-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.950422 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.950378 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:08:59.958243 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:08:59.958223 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbqz\" (UniqueName: \"kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:00.454404 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.454371 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:00.456920 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.456890 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") pod \"raw-sklearn-1b556-predictor-6bfd95dfc9-sns74\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:00.693682 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.693638 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:00.824472 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.824421 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:09:00.826689 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:09:00.826663 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bf8885_c618_4380_bbde_a36691630392.slice/crio-d9e73bcdc745a5e5cd478c16d1320bf0c7a5407161dadb7eb01694c1376bfd98 WatchSource:0}: Error finding container d9e73bcdc745a5e5cd478c16d1320bf0c7a5407161dadb7eb01694c1376bfd98: Status 404 returned error can't find the container with id d9e73bcdc745a5e5cd478c16d1320bf0c7a5407161dadb7eb01694c1376bfd98 May 20 14:09:00.971589 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.971545 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerStarted","Data":"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1"} May 20 14:09:00.971589 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:00.971586 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerStarted","Data":"d9e73bcdc745a5e5cd478c16d1320bf0c7a5407161dadb7eb01694c1376bfd98"} May 20 14:09:02.290724 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.290700 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/1.log" May 20 14:09:02.291096 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.291081 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/0.log" May 20 14:09:02.291156 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.291145 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:09:02.369005 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.368904 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert\") pod \"75adfcb7-d701-40c1-8454-d83299a69a8e\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " May 20 14:09:02.369005 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.368978 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4567h\" (UniqueName: \"kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h\") pod \"75adfcb7-d701-40c1-8454-d83299a69a8e\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " May 20 14:09:02.369252 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369012 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\") pod \"75adfcb7-d701-40c1-8454-d83299a69a8e\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " May 20 14:09:02.369252 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369052 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location\") pod \"75adfcb7-d701-40c1-8454-d83299a69a8e\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " May 20 14:09:02.369252 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369104 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") pod \"75adfcb7-d701-40c1-8454-d83299a69a8e\" (UID: \"75adfcb7-d701-40c1-8454-d83299a69a8e\") " May 20 14:09:02.369413 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369355 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75adfcb7-d701-40c1-8454-d83299a69a8e" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:09:02.369469 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369412 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config") pod "75adfcb7-d701-40c1-8454-d83299a69a8e" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e"). InnerVolumeSpecName "isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:09:02.369469 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.369415 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75adfcb7-d701-40c1-8454-d83299a69a8e" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:09:02.371337 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.371311 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75adfcb7-d701-40c1-8454-d83299a69a8e" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:09:02.371448 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.371374 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h" (OuterVolumeSpecName: "kube-api-access-4567h") pod "75adfcb7-d701-40c1-8454-d83299a69a8e" (UID: "75adfcb7-d701-40c1-8454-d83299a69a8e"). InnerVolumeSpecName "kube-api-access-4567h". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:09:02.470167 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.470129 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75adfcb7-d701-40c1-8454-d83299a69a8e-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:09:02.470167 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.470163 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:09:02.470167 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.470173 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4567h\" (UniqueName: \"kubernetes.io/projected/75adfcb7-d701-40c1-8454-d83299a69a8e-kube-api-access-4567h\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:09:02.470397 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.470185 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75adfcb7-d701-40c1-8454-d83299a69a8e-isvc-init-fail-bb9d97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:09:02.470397 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.470195 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75adfcb7-d701-40c1-8454-d83299a69a8e-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:09:02.979945 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.979918 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/1.log" May 20 14:09:02.980300 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980285 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k_75adfcb7-d701-40c1-8454-d83299a69a8e/storage-initializer/0.log" May 20 14:09:02.980367 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980321 2554 generic.go:358] "Generic (PLEG): container finished" podID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerID="bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521" exitCode=1 May 20 14:09:02.980419 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980400 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerDied","Data":"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521"} May 20 14:09:02.980419 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980412 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" May 20 14:09:02.980485 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980435 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k" event={"ID":"75adfcb7-d701-40c1-8454-d83299a69a8e","Type":"ContainerDied","Data":"c2aebeb8967f3cda08cdd6e1541bfc77ad2c24960822dca594868c45744abf86"} May 20 14:09:02.980485 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.980450 2554 scope.go:117] "RemoveContainer" containerID="bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521" May 20 14:09:02.988659 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.988641 2554 scope.go:117] "RemoveContainer" containerID="2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779" May 20 14:09:02.995973 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.995957 2554 scope.go:117] "RemoveContainer" containerID="bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521" May 20 14:09:02.996239 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:09:02.996219 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521\": container with ID starting with bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521 not found: ID does not exist" containerID="bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521" May 20 14:09:02.996301 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.996249 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521"} err="failed to get container status \"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521\": rpc error: code = NotFound desc = could not find container \"bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521\": container with ID starting with bff3564e82d8c65d4a9156017e2df98751c8b3f6773b18a6e643643a1d567521 not found: ID does not exist" May 20 14:09:02.996301 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.996277 2554 scope.go:117] "RemoveContainer" containerID="2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779" May 20 14:09:02.996552 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:09:02.996529 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779\": container with ID starting with 2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779 not found: ID does not exist" containerID="2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779" May 20 14:09:02.996595 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:02.996562 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779"} err="failed to get container status \"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779\": rpc error: code = NotFound desc = could not find container \"2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779\": container with ID starting with 2d2e25939058c9485c5315f77f38917b68e25f19e55939faa43f7540228f1779 not found: ID does not exist" May 20 14:09:03.013660 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:03.013629 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:09:03.017960 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:03.017932 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-bb9d97-predictor-64cb7bd4b5-t2p6k"] May 20 14:09:04.927694 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:04.927670 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" path="/var/lib/kubelet/pods/75adfcb7-d701-40c1-8454-d83299a69a8e/volumes" May 20 14:09:04.989286 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:04.989252 2554 generic.go:358] "Generic (PLEG): container finished" podID="b4bf8885-c618-4380-bbde-a36691630392" containerID="5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1" exitCode=0 May 20 14:09:04.989454 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:04.989323 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerDied","Data":"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1"} May 20 14:09:05.994303 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:05.994263 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerStarted","Data":"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce"} May 20 14:09:05.994303 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:05.994308 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerStarted","Data":"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698"} May 20 14:09:05.994779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:05.994594 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:05.994779 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:05.994727 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:05.996121 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:05.996096 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:06.015150 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:06.015088 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podStartSLOduration=7.01507334 podStartE2EDuration="7.01507334s" podCreationTimestamp="2026-05-20 14:08:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:06.014033919 +0000 UTC m=+1141.720465993" watchObservedRunningTime="2026-05-20 14:09:06.01507334 +0000 UTC m=+1141.721505405" May 20 14:09:06.997765 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:06.997724 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:12.002507 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:12.002480 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:09:12.003069 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:12.003040 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:22.004236 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:22.004142 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:32.003082 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:32.003041 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:42.003693 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:42.003653 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:09:52.003617 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:09:52.003569 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:10:02.003388 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:02.003354 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:10:04.912355 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.912322 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:10:04.912915 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.912336 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:10:04.941521 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.941494 2554 scope.go:117] "RemoveContainer" containerID="b05f01e8acb85ffae5fea5284ad2f4e6ebff036e57d36dc9e25dd1c695f64ab1" May 20 14:10:04.949442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.949426 2554 scope.go:117] "RemoveContainer" containerID="f68ad9e4ecbcb11d673267f826a9e6097fc089280e640385ac381397e2e3d1c3" May 20 14:10:04.956920 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.956895 2554 scope.go:117] "RemoveContainer" containerID="9b0eb757ca78aff3220435a4af0eeba08729dc46702762cc5280f5ec7e60cfb3" May 20 14:10:04.964345 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:04.964320 2554 scope.go:117] "RemoveContainer" containerID="ae5ac74c01dab31506a768a6db49fb6f2d00466fb6300ec0c613e22af56190c8" May 20 14:10:12.004118 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:12.004086 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:10:19.888704 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.888658 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:10:19.889159 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.889070 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" containerID="cri-o://7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698" gracePeriod=30 May 20 14:10:19.889159 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.889086 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kube-rbac-proxy" containerID="cri-o://bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce" gracePeriod=30 May 20 14:10:19.981947 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.981916 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:10:19.982326 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982313 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.982369 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982328 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.982369 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982351 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.982369 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982356 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.982464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982407 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.982464 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.982416 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="75adfcb7-d701-40c1-8454-d83299a69a8e" containerName="storage-initializer" May 20 14:10:19.986983 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.986964 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:19.989368 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.989347 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-83724-predictor-serving-cert\"" May 20 14:10:19.989488 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.989352 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\"" May 20 14:10:19.997975 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:19.997950 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:10:20.060558 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.060526 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.060739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.060599 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.060739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.060638 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm67s\" (UniqueName: \"kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.060739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.060668 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.060739 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.060685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161319 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161213 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vm67s\" (UniqueName: \"kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161319 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161263 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161319 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161280 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161612 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161322 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161612 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161385 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.161727 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.161679 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.162090 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.162061 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.162331 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.162242 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.164098 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.164074 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.169989 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.169959 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm67s\" (UniqueName: \"kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s\") pod \"raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.232697 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.232662 2554 generic.go:358] "Generic (PLEG): container finished" podID="b4bf8885-c618-4380-bbde-a36691630392" containerID="bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce" exitCode=2 May 20 14:10:20.232870 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.232743 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerDied","Data":"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce"} May 20 14:10:20.297872 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.297824 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:20.433917 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:20.433851 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:10:20.436792 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:10:20.436766 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d72a935_c239_42e7_8a84_0d60da5188db.slice/crio-8bfe927f446e22daf3b1824cc11778619bcfdcc44191580ed7d8f68fddfc4a71 WatchSource:0}: Error finding container 8bfe927f446e22daf3b1824cc11778619bcfdcc44191580ed7d8f68fddfc4a71: Status 404 returned error can't find the container with id 8bfe927f446e22daf3b1824cc11778619bcfdcc44191580ed7d8f68fddfc4a71 May 20 14:10:21.237863 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:21.237811 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerStarted","Data":"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba"} May 20 14:10:21.237863 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:21.237864 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerStarted","Data":"8bfe927f446e22daf3b1824cc11778619bcfdcc44191580ed7d8f68fddfc4a71"} May 20 14:10:21.998753 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:21.998706 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" May 20 14:10:22.003460 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:22.003431 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" May 20 14:10:24.249158 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:24.249126 2554 generic.go:358] "Generic (PLEG): container finished" podID="4d72a935-c239-42e7-8a84-0d60da5188db" containerID="babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba" exitCode=0 May 20 14:10:24.249566 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:24.249199 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerDied","Data":"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba"} May 20 14:10:24.930061 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:24.930032 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:10:25.109890 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.109775 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location\") pod \"b4bf8885-c618-4380-bbde-a36691630392\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " May 20 14:10:25.109890 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.109856 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbqz\" (UniqueName: \"kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz\") pod \"b4bf8885-c618-4380-bbde-a36691630392\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " May 20 14:10:25.110138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.109903 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert\") pod \"b4bf8885-c618-4380-bbde-a36691630392\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " May 20 14:10:25.110138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.109986 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-1b556-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config\") pod \"b4bf8885-c618-4380-bbde-a36691630392\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " May 20 14:10:25.110138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.110024 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") pod \"b4bf8885-c618-4380-bbde-a36691630392\" (UID: \"b4bf8885-c618-4380-bbde-a36691630392\") " May 20 14:10:25.110138 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.110115 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4bf8885-c618-4380-bbde-a36691630392" (UID: "b4bf8885-c618-4380-bbde-a36691630392"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:10:25.110368 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.110352 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-1b556-kube-rbac-proxy-sar-config") pod "b4bf8885-c618-4380-bbde-a36691630392" (UID: "b4bf8885-c618-4380-bbde-a36691630392"). InnerVolumeSpecName "raw-sklearn-1b556-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:10:25.110428 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.110366 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bf8885-c618-4380-bbde-a36691630392-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:10:25.110428 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.110393 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4bf8885-c618-4380-bbde-a36691630392" (UID: "b4bf8885-c618-4380-bbde-a36691630392"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:10:25.112511 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.112487 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4bf8885-c618-4380-bbde-a36691630392" (UID: "b4bf8885-c618-4380-bbde-a36691630392"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:10:25.112645 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.112520 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz" (OuterVolumeSpecName: "kube-api-access-4bbqz") pod "b4bf8885-c618-4380-bbde-a36691630392" (UID: "b4bf8885-c618-4380-bbde-a36691630392"). InnerVolumeSpecName "kube-api-access-4bbqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:10:25.211786 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.211756 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbqz\" (UniqueName: \"kubernetes.io/projected/b4bf8885-c618-4380-bbde-a36691630392-kube-api-access-4bbqz\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:10:25.211786 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.211785 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:10:25.212002 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.211796 2554 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-1b556-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bf8885-c618-4380-bbde-a36691630392-raw-sklearn-1b556-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:10:25.212002 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.211808 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bf8885-c618-4380-bbde-a36691630392-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:10:25.254209 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.254180 2554 generic.go:358] "Generic (PLEG): container finished" podID="b4bf8885-c618-4380-bbde-a36691630392" containerID="7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698" exitCode=0 May 20 14:10:25.254627 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.254259 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" May 20 14:10:25.254627 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.254267 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerDied","Data":"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698"} May 20 14:10:25.254627 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.254314 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74" event={"ID":"b4bf8885-c618-4380-bbde-a36691630392","Type":"ContainerDied","Data":"d9e73bcdc745a5e5cd478c16d1320bf0c7a5407161dadb7eb01694c1376bfd98"} May 20 14:10:25.254627 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.254336 2554 scope.go:117] "RemoveContainer" containerID="bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce" May 20 14:10:25.256400 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.256379 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerStarted","Data":"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15"} May 20 14:10:25.256493 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.256412 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerStarted","Data":"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524"} May 20 14:10:25.256719 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.256701 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:25.256789 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.256734 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:25.258196 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.258168 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:10:25.263716 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.263697 2554 scope.go:117] "RemoveContainer" containerID="7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698" May 20 14:10:25.271441 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.271426 2554 scope.go:117] "RemoveContainer" containerID="5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1" May 20 14:10:25.278714 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.278697 2554 scope.go:117] "RemoveContainer" containerID="bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce" May 20 14:10:25.279060 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:10:25.279037 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce\": container with ID starting with bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce not found: ID does not exist" containerID="bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce" May 20 14:10:25.279131 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.279071 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce"} err="failed to get container status \"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce\": rpc error: code = NotFound desc = could not find container \"bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce\": container with ID starting with bca16d2bf56b8d0aeda40f4490647332fb9e9f42741fd0d986cb7d81c48117ce not found: ID does not exist" May 20 14:10:25.279131 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.279089 2554 scope.go:117] "RemoveContainer" containerID="7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698" May 20 14:10:25.279348 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:10:25.279330 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698\": container with ID starting with 7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698 not found: ID does not exist" containerID="7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698" May 20 14:10:25.279413 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.279357 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698"} err="failed to get container status \"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698\": rpc error: code = NotFound desc = could not find container \"7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698\": container with ID starting with 7ed3a15dbf6afc1535da431d24e6d6eb1ec8f6b1d73f49020ab376ef79055698 not found: ID does not exist" May 20 14:10:25.279413 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.279373 2554 scope.go:117] "RemoveContainer" containerID="5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1" May 20 14:10:25.279633 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:10:25.279616 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1\": container with ID starting with 5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1 not found: ID does not exist" containerID="5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1" May 20 14:10:25.279688 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.279638 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1"} err="failed to get container status \"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1\": rpc error: code = NotFound desc = could not find container \"5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1\": container with ID starting with 5733a2fcb0bcbb53407acbc46da4135b5486e2ddabb4a4dcb83b214c42f74ba1 not found: ID does not exist" May 20 14:10:25.281753 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.281715 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podStartSLOduration=6.281703914 podStartE2EDuration="6.281703914s" podCreationTimestamp="2026-05-20 14:10:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:10:25.280435274 +0000 UTC m=+1220.986867336" watchObservedRunningTime="2026-05-20 14:10:25.281703914 +0000 UTC m=+1220.988135976" May 20 14:10:25.296751 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.296727 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:10:25.302968 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:25.302948 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b556-predictor-6bfd95dfc9-sns74"] May 20 14:10:26.261942 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:26.261895 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:10:26.927350 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:26.927318 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4bf8885-c618-4380-bbde-a36691630392" path="/var/lib/kubelet/pods/b4bf8885-c618-4380-bbde-a36691630392/volumes" May 20 14:10:31.266328 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:31.266300 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:10:31.266895 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:31.266868 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:10:41.267390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:41.267351 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:10:51.267290 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:10:51.267246 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:11:01.267284 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:01.267198 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:11:11.267856 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:11.267793 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:11:21.267262 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:21.267220 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:11:31.268272 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:31.268241 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:11:40.042445 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:40.042410 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:11:40.043001 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:40.042718 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" containerID="cri-o://e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524" gracePeriod=30 May 20 14:11:40.043001 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:40.042785 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" containerID="cri-o://3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15" gracePeriod=30 May 20 14:11:40.504225 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:40.504191 2554 generic.go:358] "Generic (PLEG): container finished" podID="4d72a935-c239-42e7-8a84-0d60da5188db" containerID="3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15" exitCode=2 May 20 14:11:40.504409 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:40.504269 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerDied","Data":"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15"} May 20 14:11:41.262943 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:41.262900 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 20 14:11:41.267249 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:41.267226 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 20 14:11:46.262346 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:46.262299 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 20 14:11:47.887021 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.886991 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:11:47.918201 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918171 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location\") pod \"4d72a935-c239-42e7-8a84-0d60da5188db\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " May 20 14:11:47.918390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918211 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls\") pod \"4d72a935-c239-42e7-8a84-0d60da5188db\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " May 20 14:11:47.918390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918261 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm67s\" (UniqueName: \"kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s\") pod \"4d72a935-c239-42e7-8a84-0d60da5188db\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " May 20 14:11:47.918390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918318 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert\") pod \"4d72a935-c239-42e7-8a84-0d60da5188db\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " May 20 14:11:47.918390 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918374 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\") pod \"4d72a935-c239-42e7-8a84-0d60da5188db\" (UID: \"4d72a935-c239-42e7-8a84-0d60da5188db\") " May 20 14:11:47.918722 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918557 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d72a935-c239-42e7-8a84-0d60da5188db" (UID: "4d72a935-c239-42e7-8a84-0d60da5188db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 20 14:11:47.918722 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918643 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d72a935-c239-42e7-8a84-0d60da5188db-kserve-provision-location\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:11:47.918994 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.918962 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config") pod "4d72a935-c239-42e7-8a84-0d60da5188db" (UID: "4d72a935-c239-42e7-8a84-0d60da5188db"). InnerVolumeSpecName "raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:11:47.919663 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.919128 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d72a935-c239-42e7-8a84-0d60da5188db" (UID: "4d72a935-c239-42e7-8a84-0d60da5188db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 20 14:11:47.921067 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.921042 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d72a935-c239-42e7-8a84-0d60da5188db" (UID: "4d72a935-c239-42e7-8a84-0d60da5188db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 20 14:11:47.921201 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:47.921065 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s" (OuterVolumeSpecName: "kube-api-access-vm67s") pod "4d72a935-c239-42e7-8a84-0d60da5188db" (UID: "4d72a935-c239-42e7-8a84-0d60da5188db"). InnerVolumeSpecName "kube-api-access-vm67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 20 14:11:48.019865 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.019733 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-cabundle-cert\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:11:48.019865 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.019771 2554 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d72a935-c239-42e7-8a84-0d60da5188db-raw-sklearn-runtime-83724-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:11:48.019865 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.019783 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d72a935-c239-42e7-8a84-0d60da5188db-proxy-tls\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:11:48.019865 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.019792 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm67s\" (UniqueName: \"kubernetes.io/projected/4d72a935-c239-42e7-8a84-0d60da5188db-kube-api-access-vm67s\") on node \"ip-10-0-140-251.ec2.internal\" DevicePath \"\"" May 20 14:11:48.529113 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.529075 2554 generic.go:358] "Generic (PLEG): container finished" podID="4d72a935-c239-42e7-8a84-0d60da5188db" containerID="e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524" exitCode=0 May 20 14:11:48.529271 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.529161 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" May 20 14:11:48.529271 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.529160 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerDied","Data":"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524"} May 20 14:11:48.529364 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.529268 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h" event={"ID":"4d72a935-c239-42e7-8a84-0d60da5188db","Type":"ContainerDied","Data":"8bfe927f446e22daf3b1824cc11778619bcfdcc44191580ed7d8f68fddfc4a71"} May 20 14:11:48.529364 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.529287 2554 scope.go:117] "RemoveContainer" containerID="3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15" May 20 14:11:48.543351 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.543331 2554 scope.go:117] "RemoveContainer" containerID="e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524" May 20 14:11:48.551024 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.550990 2554 scope.go:117] "RemoveContainer" containerID="babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba" May 20 14:11:48.557008 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.556982 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:11:48.558896 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.558880 2554 scope.go:117] "RemoveContainer" containerID="3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15" May 20 14:11:48.559227 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:11:48.559188 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15\": container with ID starting with 3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15 not found: ID does not exist" containerID="3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15" May 20 14:11:48.559316 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.559226 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15"} err="failed to get container status \"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15\": rpc error: code = NotFound desc = could not find container \"3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15\": container with ID starting with 3d5ab0fb95daf7754e1444fc5610807c68d5d72f333054222cc9e4490d6b4d15 not found: ID does not exist" May 20 14:11:48.559316 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.559253 2554 scope.go:117] "RemoveContainer" containerID="e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524" May 20 14:11:48.559777 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:11:48.559578 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524\": container with ID starting with e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524 not found: ID does not exist" containerID="e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524" May 20 14:11:48.559777 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.559644 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524"} err="failed to get container status \"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524\": rpc error: code = NotFound desc = could not find container \"e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524\": container with ID starting with e5ccbf918efe540f28c66a91c45b0031f8eb2109d894b083ac86504d6e982524 not found: ID does not exist" May 20 14:11:48.559777 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.559690 2554 scope.go:117] "RemoveContainer" containerID="babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba" May 20 14:11:48.560134 ip-10-0-140-251 kubenswrapper[2554]: E0520 14:11:48.559991 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba\": container with ID starting with babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba not found: ID does not exist" containerID="babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba" May 20 14:11:48.560134 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.560021 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba"} err="failed to get container status \"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba\": rpc error: code = NotFound desc = could not find container \"babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba\": container with ID starting with babaaaf111c4a7db8e0821c8742072b9202ee478170d4bd9ffc31d68b4df46ba not found: ID does not exist" May 20 14:11:48.561283 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.561260 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-83724-predictor-d9995568c-7jh6h"] May 20 14:11:48.927301 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:11:48.927214 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" path="/var/lib/kubelet/pods/4d72a935-c239-42e7-8a84-0d60da5188db/volumes" May 20 14:12:09.299500 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299457 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kzhb4/must-gather-z9nst"] May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299866 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299883 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299897 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="storage-initializer" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299904 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="storage-initializer" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299910 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kube-rbac-proxy" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299916 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kube-rbac-proxy" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299930 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299936 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" May 20 14:12:09.299936 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299942 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299946 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299957 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="storage-initializer" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.299962 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="storage-initializer" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.300017 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kube-rbac-proxy" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.300026 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kube-rbac-proxy" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.300032 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4bf8885-c618-4380-bbde-a36691630392" containerName="kserve-container" May 20 14:12:09.300221 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.300038 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d72a935-c239-42e7-8a84-0d60da5188db" containerName="kserve-container" May 20 14:12:09.304319 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.304301 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.307064 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.307038 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzhb4\"/\"kube-root-ca.crt\"" May 20 14:12:09.307193 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.307072 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kzhb4\"/\"default-dockercfg-spk4j\"" May 20 14:12:09.307193 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.307115 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzhb4\"/\"openshift-service-ca.crt\"" May 20 14:12:09.312669 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.312643 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzhb4/must-gather-z9nst"] May 20 14:12:09.412970 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.412927 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3648b31d-2dad-4022-a8d4-89105b8111ae-must-gather-output\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.413165 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.412994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpz5\" (UniqueName: \"kubernetes.io/projected/3648b31d-2dad-4022-a8d4-89105b8111ae-kube-api-access-7kpz5\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.514293 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.514260 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3648b31d-2dad-4022-a8d4-89105b8111ae-must-gather-output\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.514472 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.514326 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpz5\" (UniqueName: \"kubernetes.io/projected/3648b31d-2dad-4022-a8d4-89105b8111ae-kube-api-access-7kpz5\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.514626 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.514604 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3648b31d-2dad-4022-a8d4-89105b8111ae-must-gather-output\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.523773 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.523746 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpz5\" (UniqueName: \"kubernetes.io/projected/3648b31d-2dad-4022-a8d4-89105b8111ae-kube-api-access-7kpz5\") pod \"must-gather-z9nst\" (UID: \"3648b31d-2dad-4022-a8d4-89105b8111ae\") " pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.627658 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.627553 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzhb4/must-gather-z9nst" May 20 14:12:09.759009 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:09.758974 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzhb4/must-gather-z9nst"] May 20 14:12:09.762892 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:12:09.762864 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3648b31d_2dad_4022_a8d4_89105b8111ae.slice/crio-d1414a3b336fed7a01e4a596deae6b16e753f8e33d3f99683ab4b4abe7e462f5 WatchSource:0}: Error finding container d1414a3b336fed7a01e4a596deae6b16e753f8e33d3f99683ab4b4abe7e462f5: Status 404 returned error can't find the container with id d1414a3b336fed7a01e4a596deae6b16e753f8e33d3f99683ab4b4abe7e462f5 May 20 14:12:10.596700 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:10.596665 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzhb4/must-gather-z9nst" event={"ID":"3648b31d-2dad-4022-a8d4-89105b8111ae","Type":"ContainerStarted","Data":"d1414a3b336fed7a01e4a596deae6b16e753f8e33d3f99683ab4b4abe7e462f5"} May 20 14:12:11.602019 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:11.601979 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzhb4/must-gather-z9nst" event={"ID":"3648b31d-2dad-4022-a8d4-89105b8111ae","Type":"ContainerStarted","Data":"585defb46e98930b19bd1688fcc96ca278942dce9f86011572501980258c3be3"} May 20 14:12:11.602019 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:11.602020 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzhb4/must-gather-z9nst" event={"ID":"3648b31d-2dad-4022-a8d4-89105b8111ae","Type":"ContainerStarted","Data":"e017e57e377e3d9bd01ab020722db222fd57d06312a6d304f94d563d3f8545af"} May 20 14:12:12.120165 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:12.120137 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hgn5x_5bf1705e-a9b8-4b56-b283-ac216c3b2b2f/global-pull-secret-syncer/0.log" May 20 14:12:12.122671 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:12.122648 2554 ???:1] "http: TLS handshake error from 10.0.140.251:44716: EOF" May 20 14:12:12.277647 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:12.277615 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pv5cb_74cfe131-dc6a-4e79-a607-527b9b4b07f4/konnectivity-agent/0.log" May 20 14:12:12.396795 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:12.396714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-251.ec2.internal_a667969261292a12d7feb41177d18432/haproxy/0.log" May 20 14:12:15.914254 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:15.914164 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/alertmanager/0.log" May 20 14:12:15.940245 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:15.940218 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/config-reloader/0.log" May 20 14:12:15.966268 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:15.966236 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/kube-rbac-proxy-web/0.log" May 20 14:12:15.990615 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:15.990588 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/kube-rbac-proxy/0.log" May 20 14:12:16.015006 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.014966 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/kube-rbac-proxy-metric/0.log" May 20 14:12:16.035677 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.035627 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/prom-label-proxy/0.log" May 20 14:12:16.068742 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.068719 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148a0223-2d5e-4b5b-b48d-4220f0801ab2/init-config-reloader/0.log" May 20 14:12:16.178241 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.178155 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-state-metrics/0.log" May 20 14:12:16.203339 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.203306 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-rbac-proxy-main/0.log" May 20 14:12:16.228045 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.227991 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-lxmn2_b216f484-8614-4e6d-aa56-0fca48baf258/kube-rbac-proxy-self/0.log" May 20 14:12:16.259084 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.259002 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76fdb89bc9-jzzrd_4adf3390-37c1-40a6-bea0-e18d040c2db1/metrics-server/0.log" May 20 14:12:16.288630 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.288603 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-m65x7_242eeef9-b130-4cdf-91a5-431230b35128/monitoring-plugin/0.log" May 20 14:12:16.394366 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.394331 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/node-exporter/0.log" May 20 14:12:16.416371 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.416338 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/kube-rbac-proxy/0.log" May 20 14:12:16.439969 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.439885 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tp5tq_c9798965-fae2-4cee-ba67-986c20612bdd/init-textfile/0.log" May 20 14:12:16.543084 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.543055 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/kube-rbac-proxy-main/0.log" May 20 14:12:16.565466 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.565439 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/kube-rbac-proxy-self/0.log" May 20 14:12:16.592624 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.592593 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6gt5l_e556af9d-264a-476c-8625-1075d2deb558/openshift-state-metrics/0.log" May 20 14:12:16.636261 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.636227 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/prometheus/0.log" May 20 14:12:16.652703 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.652676 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/config-reloader/0.log" May 20 14:12:16.675721 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.675686 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/thanos-sidecar/0.log" May 20 14:12:16.699869 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.699779 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/kube-rbac-proxy-web/0.log" May 20 14:12:16.721732 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.721707 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/kube-rbac-proxy/0.log" May 20 14:12:16.745452 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.745426 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/kube-rbac-proxy-thanos/0.log" May 20 14:12:16.771995 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.771961 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_466e3238-d5f5-4a4d-b2b8-4601e137cb86/init-config-reloader/0.log" May 20 14:12:16.806725 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.806699 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-68fxv_54685d75-0eac-4f7a-8abc-ebbdda07ff40/prometheus-operator/0.log" May 20 14:12:16.829855 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.829811 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-68fxv_54685d75-0eac-4f7a-8abc-ebbdda07ff40/kube-rbac-proxy/0.log" May 20 14:12:16.860560 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.860522 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-96t9s_3c6d82d7-d37c-42bc-a3f0-6793334bfd0c/prometheus-operator-admission-webhook/0.log" May 20 14:12:16.898075 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.898024 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/telemeter-client/0.log" May 20 14:12:16.930363 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.930327 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/reload/0.log" May 20 14:12:16.972041 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:16.971590 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f99946db-kvw7t_066525f6-48de-4c84-b9c3-357fa78d8c2a/kube-rbac-proxy/0.log" May 20 14:12:17.022682 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.022648 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/thanos-query/0.log" May 20 14:12:17.057573 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.057534 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-web/0.log" May 20 14:12:17.088590 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.088563 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy/0.log" May 20 14:12:17.115249 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.115222 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/prom-label-proxy/0.log" May 20 14:12:17.151943 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.151895 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-rules/0.log" May 20 14:12:17.179460 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:17.179397 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b769ccb6-h2bgz_6152e737-50b6-46c0-a338-403c90da3446/kube-rbac-proxy-metrics/0.log" May 20 14:12:19.428059 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.427998 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kzhb4/must-gather-z9nst" podStartSLOduration=9.558681178 podStartE2EDuration="10.427979575s" podCreationTimestamp="2026-05-20 14:12:09 +0000 UTC" firstStartedPulling="2026-05-20 14:12:09.765094841 +0000 UTC m=+1325.471526892" lastFinishedPulling="2026-05-20 14:12:10.634393249 +0000 UTC m=+1326.340825289" observedRunningTime="2026-05-20 14:12:11.616432004 +0000 UTC m=+1327.322864066" watchObservedRunningTime="2026-05-20 14:12:19.427979575 +0000 UTC m=+1335.134411638" May 20 14:12:19.428705 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.428683 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf"] May 20 14:12:19.432403 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.432379 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.439074 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.439049 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf"] May 20 14:12:19.522101 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.522051 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-proc\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.522296 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.522119 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-podres\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.522296 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.522171 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-sys\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.522417 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.522310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjwr\" (UniqueName: \"kubernetes.io/projected/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-kube-api-access-9fjwr\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.522417 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.522377 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-lib-modules\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623450 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623416 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjwr\" (UniqueName: \"kubernetes.io/projected/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-kube-api-access-9fjwr\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623651 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623465 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-lib-modules\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623651 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623503 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-proc\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623651 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623521 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-podres\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623651 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623559 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-sys\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623651 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623642 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-proc\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623919 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623661 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-sys\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623919 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623669 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-lib-modules\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.623919 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.623680 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-podres\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.633303 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.633277 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjwr\" (UniqueName: \"kubernetes.io/projected/f3f8f9c0-1213-4098-93b5-cb3c4e4927c9-kube-api-access-9fjwr\") pod \"perf-node-gather-daemonset-5vkqf\" (UID: \"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9\") " pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.745216 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.745180 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:19.902778 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:19.902588 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf"] May 20 14:12:19.906111 ip-10-0-140-251 kubenswrapper[2554]: W0520 14:12:19.906079 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3f8f9c0_1213_4098_93b5_cb3c4e4927c9.slice/crio-85e71ba85eda76a7255d2b64bd93a25997851b5d831d131584b82265f1dab5a3 WatchSource:0}: Error finding container 85e71ba85eda76a7255d2b64bd93a25997851b5d831d131584b82265f1dab5a3: Status 404 returned error can't find the container with id 85e71ba85eda76a7255d2b64bd93a25997851b5d831d131584b82265f1dab5a3 May 20 14:12:20.255442 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.255410 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8qlkz_557d1a9e-01a7-4c3c-a75e-4414d543aaba/dns/0.log" May 20 14:12:20.278476 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.278404 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8qlkz_557d1a9e-01a7-4c3c-a75e-4414d543aaba/kube-rbac-proxy/0.log" May 20 14:12:20.367586 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.367547 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bd66_10004c10-5b3d-4c5b-a78c-9b54ce64e0a2/dns-node-resolver/0.log" May 20 14:12:20.638913 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.638797 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" event={"ID":"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9","Type":"ContainerStarted","Data":"3c53ddaa698d37bfd48b4883563013276eae5c88d72b2ad5f1d021e14e040a30"} May 20 14:12:20.638913 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.638869 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" event={"ID":"f3f8f9c0-1213-4098-93b5-cb3c4e4927c9","Type":"ContainerStarted","Data":"85e71ba85eda76a7255d2b64bd93a25997851b5d831d131584b82265f1dab5a3"} May 20 14:12:20.639421 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.639002 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:20.658939 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.658892 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" podStartSLOduration=1.658876424 podStartE2EDuration="1.658876424s" podCreationTimestamp="2026-05-20 14:12:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:12:20.656631363 +0000 UTC m=+1336.363063425" watchObservedRunningTime="2026-05-20 14:12:20.658876424 +0000 UTC m=+1336.365308485" May 20 14:12:20.829167 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:20.829135 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8pt55_3410ff62-7d89-49aa-8a17-149d4444dcc7/node-ca/0.log" May 20 14:12:21.956696 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:21.956621 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xps6t_2c96636d-79c8-41f2-bdef-3567fa77eb84/serve-healthcheck-canary/0.log" May 20 14:12:22.478726 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:22.478693 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5knr_ee6476bf-ec84-412b-ab6f-bc8ab4040ecf/kube-rbac-proxy/0.log" May 20 14:12:22.499848 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:22.499805 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5knr_ee6476bf-ec84-412b-ab6f-bc8ab4040ecf/exporter/0.log" May 20 14:12:22.523430 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:22.523401 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5knr_ee6476bf-ec84-412b-ab6f-bc8ab4040ecf/extractor/0.log" May 20 14:12:24.709285 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:24.709255 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-98dxz_5d0f2895-540e-4d6b-a200-4f03751281d7/s3-init/0.log" May 20 14:12:24.725047 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:24.725019 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-98dxz_5d0f2895-540e-4d6b-a200-4f03751281d7/download-hf-model/0.log" May 20 14:12:26.654742 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:26.654715 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kzhb4/perf-node-gather-daemonset-5vkqf" May 20 14:12:30.291716 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.291685 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/kube-multus-additional-cni-plugins/0.log" May 20 14:12:30.315077 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.315044 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/egress-router-binary-copy/0.log" May 20 14:12:30.337734 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.337669 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/cni-plugins/0.log" May 20 14:12:30.360517 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.360482 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/bond-cni-plugin/0.log" May 20 14:12:30.382411 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.382386 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/routeoverride-cni/0.log" May 20 14:12:30.414646 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.414619 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/whereabouts-cni-bincopy/0.log" May 20 14:12:30.439144 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.439116 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m28lx_eafbe133-94fb-459a-911a-0395ad08a491/whereabouts-cni/0.log" May 20 14:12:30.513742 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.513704 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pccg2_782ab188-fa6f-46cb-bac9-43fe8de7d548/kube-multus/0.log" May 20 14:12:30.533937 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.533901 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-54sgt_0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951/network-metrics-daemon/0.log" May 20 14:12:30.553653 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:30.553586 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-54sgt_0c8fcfe5-0beb-4501-81d5-f9ba0bfa0951/kube-rbac-proxy/0.log" May 20 14:12:31.781316 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.781266 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-controller/0.log" May 20 14:12:31.802885 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.802852 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/0.log" May 20 14:12:31.811437 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.811397 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovn-acl-logging/1.log" May 20 14:12:31.828808 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.828784 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/kube-rbac-proxy-node/0.log" May 20 14:12:31.854157 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.854134 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/kube-rbac-proxy-ovn-metrics/0.log" May 20 14:12:31.876882 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.876856 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/northd/0.log" May 20 14:12:31.898939 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.898918 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/nbdb/0.log" May 20 14:12:31.926717 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:31.926696 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/sbdb/0.log" May 20 14:12:32.054916 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:32.054799 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95czc_90c2ebbf-fd36-46b1-bbe1-1573a66883b4/ovnkube-controller/0.log" May 20 14:12:33.580874 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:33.580822 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tgjlf_cfd2613a-159a-4cc9-a098-cb3d1fdd0555/network-check-target-container/0.log" May 20 14:12:34.502661 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:34.502626 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mm4jd_476b93ab-c192-445a-b10f-d1a1782ef131/iptables-alerter/0.log" May 20 14:12:35.233331 ip-10-0-140-251 kubenswrapper[2554]: I0520 14:12:35.233299 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-b82vg_de5083e0-bd7d-40bc-9c80-9b7a2bdb4bed/tuned/0.log"