Jun 02 15:46:42.996665 ip-10-0-131-30 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:46:43.454050 ip-10-0-131-30 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:43.454050 ip-10-0-131-30 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:46:43.454050 ip-10-0-131-30 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:43.454050 ip-10-0-131-30 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:46:43.454050 ip-10-0-131-30 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:43.455784 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.455585    2571 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:46:43.458741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458724    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:43.458741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458740    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458744    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458747    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458750    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458753    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458756    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458764    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458767    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458770    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458772    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458775    2571 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458777    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458780    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458783    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458785    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458789    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458791    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458794    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458797    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458799    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:43.458805 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458802    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458804    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458807    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458810    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458813    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458816    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458819    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458822    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458824    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458827    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458829    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458832    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458834    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458836    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458839    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458841    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458845    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458847    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458850    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458853    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:43.459313 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458855    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458858    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458860    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458863    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458865    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458868    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458870    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458873    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458876    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458878    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458881    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458883    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458888    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458892    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458895    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458898    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458901    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458903    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458906    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:43.459857 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458908    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458911    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458913    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458916    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458919    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458921    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458923    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458926    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458929    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458931    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458935    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458938    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458941    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458944    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458946    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458950    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458952    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458955    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458957    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:43.460311 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458960    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458963    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458965    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458968    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458970    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458972    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.458975    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459406    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459413    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459418    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459421    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459424    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459428    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459431    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459433    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459436    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459439    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459442    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459445    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459447    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:43.460784 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459450    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459452    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459455    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459457    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459460    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459463    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459466    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459469    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459471    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459473    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459476    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459479    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459481    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459483    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459486    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459488    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459491    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459493    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459496    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459498    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:43.461254 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459501    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459504    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459507    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459510    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459512    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459515    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459517    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459520    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459523    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459525    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459528    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459530    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459533    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459535    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459538    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459540    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459542    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459545    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459548    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:43.461803 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459551    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459554    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459556    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459558    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459561    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459563    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459567    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459571    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459574    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459577    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459580    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459583    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459585    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459589    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459592    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459594    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459597    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459600    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459602    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:43.462259 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459605    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459607    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459610    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459612    2571 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459615    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459617    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459620    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459622    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459627    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459631    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459633    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459636    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459638    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459641    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.459644    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459713    2571 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459721    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459772    2571 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459777    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459782    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459785    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:46:43.462741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459789    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459794    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459797    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459800    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459804    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459807    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459811    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459813    2571 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459817    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459820    2571 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459823    2571 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459826    2571 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459829    2571 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459834    2571 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459837    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459840    2571 flags.go:64] FLAG: --config-dir=""
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459843    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459846    2571 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459849    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459852    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459856    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459859    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459862    2571 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459865    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:46:43.463250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459869    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459872    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459882    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459891    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459894    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459897    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459900    2571 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459903    2571 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459906    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459911    2571 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459914    2571 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459916    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459919    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459923    2571 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459927    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459930    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459933    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459936    2571 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459939    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459944    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459947    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459950    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459953    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459956    2571 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459959    2571 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:46:43.463830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459963    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459966    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459969    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459973    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459976    2571 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459979    2571 flags.go:64] FLAG: --help="false"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459982    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459985    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459988    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459991    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459995    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.459998    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460001    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460004    2571 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460007    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460010    2571 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460013    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460015    2571 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460018    2571 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460021    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460024    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460027    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460030    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460033    2571 flags.go:64] FLAG: --lock-file=""
Jun 02 15:46:43.464448 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460036    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460039    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460043    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460049    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460052    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460055    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460058    2571 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460061    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460064    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460067    2571 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460070    2571 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460075    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460080    2571 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460084    2571 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460087    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460090    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460093    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460096    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460100    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460103    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460107    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460114    2571 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460117    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460120    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:46:43.465013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460123    2571 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460126    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460131    2571 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460134    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460138    2571 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460141    2571 flags.go:64] FLAG: --port="10250"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460144    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460147    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0260d7f481281c2d7"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460150    2571 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460153    2571 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460161    2571 flags.go:64] FLAG: --register-node="true"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460164    2571 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460167    2571 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460170    2571 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460174    2571 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460176    2571 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460179    2571 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460183    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460187    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460189    2571 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460192    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460196    2571 flags.go:64] FLAG: --runonce="false"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460199    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460202    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460205    2571 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:46:43.465629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460208    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460211    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460214    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460217    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460221    2571 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460224    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460226    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460229    2571 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460232    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460236    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460238    2571 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460241    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460246    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460249    2571 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460252    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460257    2571 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460259    2571 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460263    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460266    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460269    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460272    2571 flags.go:64] FLAG: --v="2"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460276    2571 flags.go:64] FLAG: --version="false"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460280    2571 flags.go:64] FLAG: --vmodule=""
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460284    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460288    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:46:43.466230 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460416    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460420    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460424    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460428    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460431    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460433    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460436    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460439    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460441    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460444    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460446    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460449    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460452    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460455    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460457    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460460    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460462    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460465    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460467    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460470    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:43.467068 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460473    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460475    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460478    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460480    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460484    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460487    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460489    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460492    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460495    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460497    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460499    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460502    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460505    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460507    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460510    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460514    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460517    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460519    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460522    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460524    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:43.467872 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460527    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460529    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460532    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460534    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460537    2571 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460540    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460543    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460545    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460549    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460552    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460555    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460558    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460561    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460564    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460567    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460570    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460574    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460576    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460579    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460581    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:43.468570 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460583    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460586    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460588    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460591    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460593    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460596    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460599    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460603    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460606    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460609    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460611    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460614    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460617    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460620    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460622    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460625    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460627    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460630    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460633    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460637    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:43.469116 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460641    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460644    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460646    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460649    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460651    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.460654    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:43.469637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.460659    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.469635    2571 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.469655    2571 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469726    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469732    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469736    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469739    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469742    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469745    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469749    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469751    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469754    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469757    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469760    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469762    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469765    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469768    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469771    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469773    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469776    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:43.469791 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469778    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469781    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469783    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469786    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469788    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469791    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469794    2571 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469797    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469800    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469803    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469806    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469809    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469811    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469814    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469816    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469819    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469821    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469824    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469826    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469829    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:43.470271 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469832    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469836    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469840    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469844    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469848    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469852    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469854    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469857    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469859    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469862    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469864    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469866    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469869    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469871    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469875    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469878    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469881    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469883    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469886    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:43.470772 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469889    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469892    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469895    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469897    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469900    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469903    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469906    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469908    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469911    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469913    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469916    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469919    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469921    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469924    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469926    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469929    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469931    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469934    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469936    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469939    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:43.471281 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469942    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469945    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469947    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469950    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469952    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469955    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469958    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469962    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469964    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.469967    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.469972    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470075    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470080    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470083    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470086    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470088    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:43.471806 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470092    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470094    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470097    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470099    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470102    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470105    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470107    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470110    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470112    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470115    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470117    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470119    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470122    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470125    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470128    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470130    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470133    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470136    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470138    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:43.472208 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470141    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470144    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470147    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470150    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470152    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470155    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470158    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470162    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470166    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470169    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470172    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470175    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470178    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470181    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470183    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470186    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470188    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470191    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470193    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470196    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:43.472684 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470198    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470200    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470203    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470205    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470208    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470210    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470212    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470215    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470217    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470220    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470222    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470226    2571 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470228    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470231    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470233    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470236    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470239    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470242    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470245    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:43.473173 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470249    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470251    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470254    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470257    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470259    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470262    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470264    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470267    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470269    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470272    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470275    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470278    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470280    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470283    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470285    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470288    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470290    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470293    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470295    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470298    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:43.473657 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470300    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:43.474151 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470303    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:43.474151 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:43.470305    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:43.474151 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.470310    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:43.474151 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.471199    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:46:43.474151 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.473351    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:46:43.474429 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.474416    2571 server.go:1019] "Starting client certificate rotation"
Jun 02 15:46:43.474530 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.474516    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:43.474562 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.474553    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:43.497659 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.497635    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:43.501598 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.501084    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:43.512361 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.512337    2571 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:46:43.518780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.518760    2571 log.go:25] "Validated CRI v1 image API"
Jun 02 15:46:43.520171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.520154    2571 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:46:43.524418 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.524386    2571 fs.go:135] Filesystem UUIDs: map[76f5ac24-dd1d-452f-87fc-ace78bcc9dbb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8b6389ba-1eb2-45c7-8b23-1c8f9a3fb96d:/dev/nvme0n1p4]
Jun 02 15:46:43.524418 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.524410    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:46:43.526013 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.525978    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:43.530099 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.529978    2571 manager.go:217] Machine: {Timestamp:2026-06-02 15:46:43.529465755 +0000 UTC m=+0.404991907 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098679 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fab81ed7142c30176984551175c17 SystemUUID:ec2fab81-ed71-42c3-0176-984551175c17 BootID:08f17a23-c719-4c74-94cb-bcc917530a9b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d1:98:3c:be:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d1:98:3c:be:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:1d:c1:20:98:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:46:43.530907 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.530894    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:46:43.531008 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.530996    2571 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:46:43.533209 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.533177    2571 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:46:43.533352 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.533214    2571 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:46:43.533421 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.533362    2571 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:46:43.533421 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.533384    2571 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:46:43.533421 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.533391    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:43.534339 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.534328    2571 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:43.535872 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.535861    2571 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:43.535980 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.535971    2571 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:46:43.538596 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.538587    2571 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:46:43.538634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.538600    2571 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:46:43.538634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.538615    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:46:43.538634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.538624    2571 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:46:43.538634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.538632    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:46:43.539860 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.539846    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:43.539927 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.539866    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:43.543390 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.543360    2571 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:46:43.545203 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.545188    2571 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:46:43.546973 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.546958    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:46:43.546973 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.546975    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.546982    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.546988    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.546997    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547003    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547009    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547014    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547021    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547028    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547042    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:46:43.547071 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.547048    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:46:43.548680 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.548667    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:46:43.548720 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.548682    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:46:43.552082 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.552067    2571 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:46:43.552215 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.552107    2571 server.go:1295] "Started kubelet"
Jun 02 15:46:43.552215 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.552163    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:46:43.552286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.552202    2571 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:46:43.552286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.552267    2571 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:46:43.552916 ip-10-0-131-30 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:46:43.553577 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.553563    2571 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:46:43.554209 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.554181    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:46:43.554310 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.554194    2571 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:46:43.554310 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.554218    2571 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:46:43.554310 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.554223    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:46:43.557734 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.557711    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:43.558128 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558104    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:46:43.558201 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558176    2571 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:46:43.558201 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558188    2571 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:46:43.558286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558231    2571 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:46:43.558343 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558308    2571 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:46:43.558343 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.558318    2571 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:46:43.559032 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.558093    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-30.ec2.internal.18b54e31d06d364d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-30.ec2.internal,UID:ip-10-0-131-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-30.ec2.internal,},FirstTimestamp:2026-06-02 15:46:43.552081485 +0000 UTC m=+0.427607638,LastTimestamp:2026-06-02 15:46:43.552081485 +0000 UTC m=+0.427607638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-30.ec2.internal,}"
Jun 02 15:46:43.559152 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.559096    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.559350 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559330    2571 factory.go:55] Registering systemd factory
Jun 02 15:46:43.559441 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559357    2571 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:46:43.559670 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559653    2571 factory.go:153] Registering CRI-O factory
Jun 02 15:46:43.559743 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559673    2571 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:46:43.559743 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559732    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:46:43.559842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559751    2571 factory.go:103] Registering Raw factory
Jun 02 15:46:43.559842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.559767    2571 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:46:43.560844 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.560827    2571 manager.go:319] Starting recovery of all containers
Jun 02 15:46:43.562867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.562841    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xjk5r"
Jun 02 15:46:43.563035 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.562981    2571 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:46:43.569503 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.569471    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 02 15:46:43.569627 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.569596    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 02 15:46:43.570750 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.570726    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xjk5r"
Jun 02 15:46:43.572064 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.572050    2571 manager.go:324] Recovery completed
Jun 02 15:46:43.576209 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.576198    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.578570 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.578552    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.578654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.578583    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.578654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.578592    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.579124 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.579109    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:46:43.579183 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.579125    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:46:43.579183 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.579140    2571 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:43.580634 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.580546    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-30.ec2.internal.18b54e31d201632e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-30.ec2.internal,UID:ip-10-0-131-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-30.ec2.internal,},FirstTimestamp:2026-06-02 15:46:43.578569518 +0000 UTC m=+0.454095670,LastTimestamp:2026-06-02 15:46:43.578569518 +0000 UTC m=+0.454095670,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-30.ec2.internal,}"
Jun 02 15:46:43.581457 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.581444    2571 policy_none.go:49] "None policy: Start"
Jun 02 15:46:43.581508 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.581462    2571 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:46:43.581508 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.581473    2571 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621133    2571 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.621170    2571 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621180    2571 server.go:85] "Starting device plugin registration server"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621508    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621523    2571 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621616    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621696    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.621704    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.622135    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:46:43.622573 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.622171    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.677586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.677536    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:46:43.678848 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.678828    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:46:43.678917 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.678862    2571 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:46:43.678917 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.678886    2571 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:46:43.678917 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.678896    2571 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:46:43.679042 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.679007    2571 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:46:43.681641 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.681619    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:43.722700 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.722673    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.723722 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.723706    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.723790 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.723737    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.723790 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.723748    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.723790 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.723773    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.732143 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.732126    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.732193 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.732152    2571 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-30.ec2.internal\": node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.750929 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.750902    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.779679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.779647    2571 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"]
Jun 02 15:46:43.779738 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.779727    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.780735 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.780720    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.780801 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.780749    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.780801 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.780759    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.782047 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782034    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.782216 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782201    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.782251 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782233    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.782865 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782846    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.782954 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782872    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.782954 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782882    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.782954 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782848    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.782954 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782942    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.782954 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.782953    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.784002 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.783987    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.784094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.784010    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:43.784675 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.784659    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:43.784758 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.784686    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:43.784758 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.784699    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:43.809252 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.809223    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-30.ec2.internal\" not found" node="ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.813625 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.813605    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-30.ec2.internal\" not found" node="ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.851245 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.851205    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.861295 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.861266    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.861432 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.861300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.861432 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.861318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3240da12bbab1be9fe19581d21610356-config\") pod \"kube-apiserver-proxy-ip-10-0-131-30.ec2.internal\" (UID: \"3240da12bbab1be9fe19581d21610356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.952279 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:43.952244    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:43.961703 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961676    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3240da12bbab1be9fe19581d21610356-config\") pod \"kube-apiserver-proxy-ip-10-0-131-30.ec2.internal\" (UID: \"3240da12bbab1be9fe19581d21610356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.961768 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961709    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.961768 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961727    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.961842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961770    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3240da12bbab1be9fe19581d21610356-config\") pod \"kube-apiserver-proxy-ip-10-0-131-30.ec2.internal\" (UID: \"3240da12bbab1be9fe19581d21610356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.961842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961791    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:43.961842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:43.961772    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5b9144fc1c1a59765ef243be3273ec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal\" (UID: \"5c5b9144fc1c1a59765ef243be3273ec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:44.053183 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.053104    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.112609 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.112567    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:44.117594 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.117576    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:44.154163 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.154133    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.254629 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.254604    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.355144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.355074    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.455598 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.455556    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.474007 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.473981    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:46:44.474141 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.474126    2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:44.556398 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.556218    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.558384 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.558354    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:44.573250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.573219    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:41:43 +0000 UTC" deadline="2028-03-11 07:29:43.360153796 +0000 UTC"
Jun 02 15:46:44.573250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.573244    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15543h42m58.786911994s"
Jun 02 15:46:44.573844 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.573828    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:44.602231 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:44.602201    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5b9144fc1c1a59765ef243be3273ec.slice/crio-cb52c4d09017413a6349daeccb42277bd6027dcbaad4b2d1d1891cfd36140762 WatchSource:0}: Error finding container cb52c4d09017413a6349daeccb42277bd6027dcbaad4b2d1d1891cfd36140762: Status 404 returned error can't find the container with id cb52c4d09017413a6349daeccb42277bd6027dcbaad4b2d1d1891cfd36140762
Jun 02 15:46:44.602460 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:44.602441    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3240da12bbab1be9fe19581d21610356.slice/crio-826429dd99a19d2c17699b7320e3aa8af8aee7d7029bef2e4f093db0684f150f WatchSource:0}: Error finding container 826429dd99a19d2c17699b7320e3aa8af8aee7d7029bef2e4f093db0684f150f: Status 404 returned error can't find the container with id 826429dd99a19d2c17699b7320e3aa8af8aee7d7029bef2e4f093db0684f150f
Jun 02 15:46:44.604391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.604348    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-szplp"
Jun 02 15:46:44.605641 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.605627    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:46:44.613701 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.613680    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-szplp"
Jun 02 15:46:44.656996 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.656956    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.682191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.682142    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal" event={"ID":"5c5b9144fc1c1a59765ef243be3273ec","Type":"ContainerStarted","Data":"cb52c4d09017413a6349daeccb42277bd6027dcbaad4b2d1d1891cfd36140762"}
Jun 02 15:46:44.683083 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.683059    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal" event={"ID":"3240da12bbab1be9fe19581d21610356","Type":"ContainerStarted","Data":"826429dd99a19d2c17699b7320e3aa8af8aee7d7029bef2e4f093db0684f150f"}
Jun 02 15:46:44.757277 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.757249    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.823072 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.823042    2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:44.858026 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.857957    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:44.923635 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:44.923607    2571 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:44.958845 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:44.958811    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-131-30.ec2.internal\" not found"
Jun 02 15:46:45.012713 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.012679    2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:45.058983 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.058948    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:45.071527 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.071494    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:45.072530 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.072515    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal"
Jun 02 15:46:45.144772 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.144694    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:45.409410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.409319    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:45.540034 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.540000    2571 apiserver.go:52] "Watching apiserver"
Jun 02 15:46:45.546436 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.546409    2571 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:46:45.550171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.550147    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xzf5d","openshift-multus/multus-gbcbz","openshift-multus/network-metrics-daemon-9qqr9","openshift-network-diagnostics/network-check-target-wmkbn","kube-system/konnectivity-agent-kj5mf","openshift-image-registry/node-ca-wjg4n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal","openshift-network-operator/iptables-alerter-b5lv9","openshift-ovn-kubernetes/ovnkube-node-9st6m","kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh","openshift-cluster-node-tuning-operator/tuned-7wbzp","openshift-dns/node-resolver-r4wc4"]
Jun 02 15:46:45.552141 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.552122    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.553493 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.553473    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.554700 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.554680    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.554807 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.554665    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbx5n\""
Jun 02 15:46:45.554807 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.554691    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.555401 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.555382    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.555616 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.555586    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:46:45.555709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.555615    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:46:45.555709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.555639    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.556144 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.555886    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.556144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.555967    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:45.556144 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.556006    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:45.556144 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.556054    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6wggr\""
Jun 02 15:46:45.556144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.556108    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:45.557331 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.557311    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.558681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.558661    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.559506 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.559486    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:46:45.559663 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.559649    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:46:45.559939 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.559885    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55ljc\""
Jun 02 15:46:45.561123 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.561101    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:46:45.561209 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.561141    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.561901 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.561882    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.561987 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.561972    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rjt4s\""
Jun 02 15:46:45.562792 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.562775    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.564535 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.564515    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.565131 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.565114    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:46:45.566012 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.565994    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9w56j\""
Jun 02 15:46:45.566098 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.566003    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:46:45.566516 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.566502    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.566704 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.566587    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.567666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.567616    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.567666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.567622    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gqlm4\""
Jun 02 15:46:45.567666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.567663    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.568017 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.567998    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.568677 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.568602    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:46:45.569311 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.569289    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.570066 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570049    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.570467 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570449    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.570590 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570571    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:46:45.570842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.570842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570814    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7gw\" (UniqueName: \"kubernetes.io/projected/3e5af621-7c9a-437e-9195-e97b1fa0d25e-kube-api-access-hn7gw\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.570971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570850    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.570971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570853    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-bin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.570971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570864    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:46:45.570971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-conf-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.570971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-multus-daemon-config\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.570980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqsw9\" (UniqueName: \"kubernetes.io/projected/51da022a-4320-403f-bee1-05113613afb4-kube-api-access-hqsw9\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-cni-binary-copy\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-modprobe-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-kubernetes\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-conf\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-run\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-sys\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571197 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-cnibin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9111172-9d4c-4285-9e39-ba0c9a637533-agent-certs\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9111172-9d4c-4285-9e39-ba0c9a637533-konnectivity-ca\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-systemd\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571311    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-tmp\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571335    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-os-release\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-socket-dir-parent\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-k8s-cni-cncf-io\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysconfig\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-hostroot\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m2nz\" (UniqueName: \"kubernetes.io/projected/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-kube-api-access-9m2nz\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-var-lib-kubelet\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.571586 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-host\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-tuned\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-netns\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-multus\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-kubelet\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571676    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-multus-certs\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-etc-kubernetes\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1a921c-9997-4cd6-937f-a1a1a2967260-host\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-system-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1a921c-9997-4cd6-937f-a1a1a2967260-serviceca\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxq6\" (UniqueName: \"kubernetes.io/projected/6e1a921c-9997-4cd6-937f-a1a1a2967260-kube-api-access-9qxq6\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-lib-modules\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.572094 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.571975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:45.572610 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.572499    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:45.572972 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.572775    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:46:45.572972 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.572790    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bhsjt\""
Jun 02 15:46:45.572972 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.572802    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mmw6\""
Jun 02 15:46:45.572972 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.572817    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:46:45.573265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.573077    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:46:45.573265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.573170    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5jf5\""
Jun 02 15:46:45.614382 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.614333    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:44 +0000 UTC" deadline="2027-12-01 18:08:08.881807743 +0000 UTC"
Jun 02 15:46:45.614503 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.614383    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13130h21m23.267429334s"
Jun 02 15:46:45.659908 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.659832    2571 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:46:45.672607 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672575    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysconfig\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-device-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672668    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-var-lib-kubelet\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672670    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysconfig\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672692    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-host\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672716    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-tuned\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672732    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-kubelet\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672758    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-var-lib-kubelet\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.672778 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672761    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-multus-certs\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672808    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-multus-certs\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672817    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-ovn\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672808    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-kubelet\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672818    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-host\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672849    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-system-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-system-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-node-log\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-os-release\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.672984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-bin\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-config\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/631ad240-3383-480d-ab82-bde14902804e-ovn-node-metrics-cert\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673061    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1a921c-9997-4cd6-937f-a1a1a2967260-serviceca\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673102    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxq6\" (UniqueName: \"kubernetes.io/projected/6e1a921c-9997-4cd6-937f-a1a1a2967260-kube-api-access-9qxq6\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673148    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvns9\" (UniqueName: \"kubernetes.io/projected/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-kube-api-access-tvns9\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673099    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:46:45.673212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-slash\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673224    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7gw\" (UniqueName: \"kubernetes.io/projected/3e5af621-7c9a-437e-9195-e97b1fa0d25e-kube-api-access-hn7gw\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673251    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-bin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673276    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-multus-daemon-config\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673317    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hqsw9\" (UniqueName: \"kubernetes.io/projected/51da022a-4320-403f-bee1-05113613afb4-kube-api-access-hqsw9\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-system-cni-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673417    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-systemd-units\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673468    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-cni-binary-copy\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673522    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e1a921c-9997-4cd6-937f-a1a1a2967260-serviceca\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.673903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673608    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-bin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673948    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-modprobe-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.673990    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-conf\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674023    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-k8s-cni-cncf-io\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674040    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9111172-9d4c-4285-9e39-ba0c9a637533-agent-certs\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674052    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-modprobe-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674059    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-hostroot\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-etc-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674090    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-k8s-cni-cncf-io\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-kubelet\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674129    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-hostroot\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674134    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674167    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-conf\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-iptables-alerter-script\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqg9\" (UniqueName: \"kubernetes.io/projected/a71f5380-9081-4609-8da0-42da280a618d-kube-api-access-glqg9\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.674223    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674235    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.674446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674278    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea971506-fe66-418b-a884-855d031a66aa-tmp-dir\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq74d\" (UniqueName: \"kubernetes.io/projected/6edf4f75-5e8d-416a-b930-c64d36a81a02-kube-api-access-rq74d\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674644    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-netns\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674679    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-etc-kubernetes\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-var-lib-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674895    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1a921c-9997-4cd6-937f-a1a1a2967260-host\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.674924    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-netns\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675171 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-script-lib\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-sys-fs\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-env-overrides\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/ea971506-fe66-418b-a884-855d031a66aa-kube-api-access-jvx66\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675336    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-lib-modules\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675393    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-registration-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675467    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675509    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-conf-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-host-slash\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-systemd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-netd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675624    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-cni-binary-copy\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.675692 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52qj\" (UniqueName: \"kubernetes.io/projected/631ad240-3383-480d-ab82-bde14902804e-kube-api-access-r52qj\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.676389 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.675720    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:46.17569607 +0000 UTC m=+3.051222221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:45.676389 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.675794    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea971506-fe66-418b-a884-855d031a66aa-hosts-file\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.676389 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51da022a-4320-403f-bee1-05113613afb4-multus-daemon-config\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.676389 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676208    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-etc-kubernetes\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.676389 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676329    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-cni-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.676618 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676595    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-run-netns\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677056 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676858    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-sysctl-d\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.677056 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.676984    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-lib-modules\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.677201 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677081    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-conf-dir\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677416 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677326    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-multus\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677416 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677330    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e1a921c-9997-4cd6-937f-a1a1a2967260-host\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.677416 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677400    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-host-var-lib-cni-multus\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677677 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677654    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-kubernetes\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.677754 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677720    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-run\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.677817 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677757    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-sys\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.677817 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677803    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-cnibin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677924 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677836    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-os-release\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.677924 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-socket-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.677924 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677909    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9111172-9d4c-4285-9e39-ba0c9a637533-konnectivity-ca\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.678065 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.677996    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-systemd\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.678065 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678031    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-tmp\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.678181 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678066    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-socket-dir-parent\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.678181 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678110    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9m2nz\" (UniqueName: \"kubernetes.io/projected/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-kube-api-access-9m2nz\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.678181 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-cnibin\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.678320 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-log-socket\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.678387 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678343    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9111172-9d4c-4285-9e39-ba0c9a637533-agent-certs\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.678527 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678504    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-sys\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.678527 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678532    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-run\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.678741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678568    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-cnibin\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.678741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678587    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-kubernetes\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.678741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-os-release\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.678741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678682    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51da022a-4320-403f-bee1-05113613afb4-multus-socket-dir-parent\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.678741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678692    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-systemd\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.679044 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.678783    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9111172-9d4c-4285-9e39-ba0c9a637533-konnectivity-ca\") pod \"konnectivity-agent-kj5mf\" (UID: \"d9111172-9d4c-4285-9e39-ba0c9a637533\") " pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.680521 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.680482    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-etc-tuned\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.681575 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.681555    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e5af621-7c9a-437e-9195-e97b1fa0d25e-tmp\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.683413 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.683387    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7gw\" (UniqueName: \"kubernetes.io/projected/3e5af621-7c9a-437e-9195-e97b1fa0d25e-kube-api-access-hn7gw\") pod \"tuned-7wbzp\" (UID: \"3e5af621-7c9a-437e-9195-e97b1fa0d25e\") " pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.683685 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.683665    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqsw9\" (UniqueName: \"kubernetes.io/projected/51da022a-4320-403f-bee1-05113613afb4-kube-api-access-hqsw9\") pod \"multus-gbcbz\" (UID: \"51da022a-4320-403f-bee1-05113613afb4\") " pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.684192 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.684174    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxq6\" (UniqueName: \"kubernetes.io/projected/6e1a921c-9997-4cd6-937f-a1a1a2967260-kube-api-access-9qxq6\") pod \"node-ca-wjg4n\" (UID: \"6e1a921c-9997-4cd6-937f-a1a1a2967260\") " pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.685422 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.685406    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:45.685500 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.685425    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:45.685500 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.685438    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:45.685591 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:45.685534    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:46:46.185515537 +0000 UTC m=+3.061041691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:45.687859 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.687841    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m2nz\" (UniqueName: \"kubernetes.io/projected/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-kube-api-access-9m2nz\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:45.778503 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778476    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-registration-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.778503 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778504    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-host-slash\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778519    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-systemd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778539    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-netd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778585    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r52qj\" (UniqueName: \"kubernetes.io/projected/631ad240-3383-480d-ab82-bde14902804e-kube-api-access-r52qj\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778604    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea971506-fe66-418b-a884-855d031a66aa-hosts-file\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778615    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-systemd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778625    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-host-slash\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778640    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-netd\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778634    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-socket-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778693    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea971506-fe66-418b-a884-855d031a66aa-hosts-file\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.778709 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778691    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-cnibin\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778739    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-log-socket\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778748    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-registration-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778766    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778792    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-log-socket\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778802    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-device-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778819    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-socket-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778835    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-ovn\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778842    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-device-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778803    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778818    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-cnibin\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778886    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-ovn\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778911    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778945    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-node-log\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778960    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-os-release\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778982    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-bin\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.778998    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-config\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779152 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779013    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/631ad240-3383-480d-ab82-bde14902804e-ovn-node-metrics-cert\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779029    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tvns9\" (UniqueName: \"kubernetes.io/projected/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-kube-api-access-tvns9\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779043    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779055    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-cni-bin\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779072    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-node-log\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779082    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-slash\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779072    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-os-release\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779121    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-slash\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779119    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-system-cni-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779152    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-system-cni-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779155    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779126    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779185    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779215    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-systemd-units\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779247    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-etc-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779268    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a71f5380-9081-4609-8da0-42da280a618d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779274    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-kubelet\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.779985 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779318    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-iptables-alerter-script\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779343    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-glqg9\" (UniqueName: \"kubernetes.io/projected/a71f5380-9081-4609-8da0-42da280a618d-kube-api-access-glqg9\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779349    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-kubelet\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779349    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-systemd-units\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779388    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779418    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea971506-fe66-418b-a884-855d031a66aa-tmp-dir\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779444    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779444    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rq74d\" (UniqueName: \"kubernetes.io/projected/6edf4f75-5e8d-416a-b930-c64d36a81a02-kube-api-access-rq74d\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779493    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-var-lib-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779517    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779576    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-netns\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779602    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-script-lib\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779613    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-var-lib-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779626    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779651    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-sys-fs\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779667    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779684    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.780681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779702    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-config\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-run-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779722    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779713    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-env-overrides\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779781    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/ea971506-fe66-418b-a884-855d031a66aa-kube-api-access-jvx66\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779798    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6edf4f75-5e8d-416a-b930-c64d36a81a02-sys-fs\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779756    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-etc-openvswitch\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779778    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779859    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/631ad240-3383-480d-ab82-bde14902804e-host-run-netns\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779904    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-iptables-alerter-script\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.779956    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea971506-fe66-418b-a884-855d031a66aa-tmp-dir\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.780144    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-ovnkube-script-lib\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.780153    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/631ad240-3383-480d-ab82-bde14902804e-env-overrides\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.781391 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.780233    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a71f5380-9081-4609-8da0-42da280a618d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.782026 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.782005    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/631ad240-3383-480d-ab82-bde14902804e-ovn-node-metrics-cert\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.788781 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.788754    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq74d\" (UniqueName: \"kubernetes.io/projected/6edf4f75-5e8d-416a-b930-c64d36a81a02-kube-api-access-rq74d\") pod \"aws-ebs-csi-driver-node-clkrh\" (UID: \"6edf4f75-5e8d-416a-b930-c64d36a81a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:45.788912 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.788828    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/ea971506-fe66-418b-a884-855d031a66aa-kube-api-access-jvx66\") pod \"node-resolver-r4wc4\" (UID: \"ea971506-fe66-418b-a884-855d031a66aa\") " pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.789028 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.788973    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvns9\" (UniqueName: \"kubernetes.io/projected/3b4a915b-6de0-4eac-8cf7-a4d48060b2e4-kube-api-access-tvns9\") pod \"iptables-alerter-b5lv9\" (UID: \"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4\") " pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.789311 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.789290    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqg9\" (UniqueName: \"kubernetes.io/projected/a71f5380-9081-4609-8da0-42da280a618d-kube-api-access-glqg9\") pod \"multus-additional-cni-plugins-xzf5d\" (UID: \"a71f5380-9081-4609-8da0-42da280a618d\") " pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.789387 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.789325    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52qj\" (UniqueName: \"kubernetes.io/projected/631ad240-3383-480d-ab82-bde14902804e-kube-api-access-r52qj\") pod \"ovnkube-node-9st6m\" (UID: \"631ad240-3383-480d-ab82-bde14902804e\") " pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.863117 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.863076    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wbzp"
Jun 02 15:46:45.872800 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.872773    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gbcbz"
Jun 02 15:46:45.880455 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.880430    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:46:45.886054 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.886031    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjg4n"
Jun 02 15:46:45.892630 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.892610    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xzf5d"
Jun 02 15:46:45.899095 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.899077    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5lv9"
Jun 02 15:46:45.905992 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.905699    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:46:45.912544 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.912487    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r4wc4"
Jun 02 15:46:45.918067 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:45.918050    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh"
Jun 02 15:46:46.182277 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.182198    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:46.182459 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.182357    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:46.182521 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.182465    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:47.1824312 +0000 UTC m=+4.057957348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:46.254678 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.254632    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9111172_9d4c_4285_9e39_ba0c9a637533.slice/crio-2d04dfcd27cc9447bf762cf5fcc3b50313c041cf957700cdda92a573a201d764 WatchSource:0}: Error finding container 2d04dfcd27cc9447bf762cf5fcc3b50313c041cf957700cdda92a573a201d764: Status 404 returned error can't find the container with id 2d04dfcd27cc9447bf762cf5fcc3b50313c041cf957700cdda92a573a201d764
Jun 02 15:46:46.255505 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.255482    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edf4f75_5e8d_416a_b930_c64d36a81a02.slice/crio-fdb3df4868d7f48add4678be0e54ce0555e7856a9dac54e14d99bed82d539609 WatchSource:0}: Error finding container fdb3df4868d7f48add4678be0e54ce0555e7856a9dac54e14d99bed82d539609: Status 404 returned error can't find the container with id fdb3df4868d7f48add4678be0e54ce0555e7856a9dac54e14d99bed82d539609
Jun 02 15:46:46.256141 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.256119    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631ad240_3383_480d_ab82_bde14902804e.slice/crio-adbd12edda966fd18f072945f67eeff4cadad2214341867847313c8884e2dce0 WatchSource:0}: Error finding container adbd12edda966fd18f072945f67eeff4cadad2214341867847313c8884e2dce0: Status 404 returned error can't find the container with id adbd12edda966fd18f072945f67eeff4cadad2214341867847313c8884e2dce0
Jun 02 15:46:46.257485 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.257276    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a915b_6de0_4eac_8cf7_a4d48060b2e4.slice/crio-bf5dc243f0a8b3332ee0f82331dba102d441198c2849bc2fbfc4218dc6cc392c WatchSource:0}: Error finding container bf5dc243f0a8b3332ee0f82331dba102d441198c2849bc2fbfc4218dc6cc392c: Status 404 returned error can't find the container with id bf5dc243f0a8b3332ee0f82331dba102d441198c2849bc2fbfc4218dc6cc392c
Jun 02 15:46:46.260082 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.260060    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51da022a_4320_403f_bee1_05113613afb4.slice/crio-3bcdfdb5e4b0aa43c0fca9ee55d35afbbd56e0eaff82ed2cfa6fda8e13786b28 WatchSource:0}: Error finding container 3bcdfdb5e4b0aa43c0fca9ee55d35afbbd56e0eaff82ed2cfa6fda8e13786b28: Status 404 returned error can't find the container with id 3bcdfdb5e4b0aa43c0fca9ee55d35afbbd56e0eaff82ed2cfa6fda8e13786b28
Jun 02 15:46:46.261832 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:46:46.261645    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5af621_7c9a_437e_9195_e97b1fa0d25e.slice/crio-fdebea8d07f3ecab0e17d939389f609695219d32304ab7829a843a8633716bd9 WatchSource:0}: Error finding container fdebea8d07f3ecab0e17d939389f609695219d32304ab7829a843a8633716bd9: Status 404 returned error can't find the container with id fdebea8d07f3ecab0e17d939389f609695219d32304ab7829a843a8633716bd9
Jun 02 15:46:46.283504 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.283484    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:46.283652 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.283635    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:46.283685 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.283655    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:46.283685 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.283664    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:46.283748 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:46.283704    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:46:47.283691402 +0000 UTC m=+4.159217542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:46.615261 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.615214    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:44 +0000 UTC" deadline="2027-12-26 23:17:52.614165625 +0000 UTC"
Jun 02 15:46:46.615261 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.615251    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13735h31m5.998916502s"
Jun 02 15:46:46.687820 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.687640    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjg4n" event={"ID":"6e1a921c-9997-4cd6-937f-a1a1a2967260","Type":"ContainerStarted","Data":"eadfcbbd9f19cd62cc656207bfdfb245f0c7e0dabc330541885f04c82dfa6ac5"}
Jun 02 15:46:46.689687 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.689618    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r4wc4" event={"ID":"ea971506-fe66-418b-a884-855d031a66aa","Type":"ContainerStarted","Data":"245655d1d44955501a8d1efedd793888d1c842736a923b46794b6bc2ce4a1393"}
Jun 02 15:46:46.691202 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.691162    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerStarted","Data":"1b47e5f931665a15d10d8ef5b97b0d3ceb5beccae59c9a689fad1a7a66552242"}
Jun 02 15:46:46.692273 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.692246    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gbcbz" event={"ID":"51da022a-4320-403f-bee1-05113613afb4","Type":"ContainerStarted","Data":"3bcdfdb5e4b0aa43c0fca9ee55d35afbbd56e0eaff82ed2cfa6fda8e13786b28"}
Jun 02 15:46:46.695069 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.694141    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5lv9" event={"ID":"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4","Type":"ContainerStarted","Data":"bf5dc243f0a8b3332ee0f82331dba102d441198c2849bc2fbfc4218dc6cc392c"}
Jun 02 15:46:46.696446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.696194    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"adbd12edda966fd18f072945f67eeff4cadad2214341867847313c8884e2dce0"}
Jun 02 15:46:46.697802 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.697753    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh" event={"ID":"6edf4f75-5e8d-416a-b930-c64d36a81a02","Type":"ContainerStarted","Data":"fdb3df4868d7f48add4678be0e54ce0555e7856a9dac54e14d99bed82d539609"}
Jun 02 15:46:46.700494 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.699766    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal" event={"ID":"3240da12bbab1be9fe19581d21610356","Type":"ContainerStarted","Data":"b6b27eb21653055999d9ee5a4ebf0df0ddc0899a86f25d05f43cc822a8c63fe6"}
Jun 02 15:46:46.705799 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.705194    2571 generic.go:358] "Generic (PLEG): container finished" podID="5c5b9144fc1c1a59765ef243be3273ec" containerID="b70c34daf037f27166487965e0814b762dd9925ca2a97c4f9d8dce2c1adcec50" exitCode=0
Jun 02 15:46:46.705799 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.705263    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal" event={"ID":"5c5b9144fc1c1a59765ef243be3273ec","Type":"ContainerDied","Data":"b70c34daf037f27166487965e0814b762dd9925ca2a97c4f9d8dce2c1adcec50"}
Jun 02 15:46:46.709070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.709040    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wbzp" event={"ID":"3e5af621-7c9a-437e-9195-e97b1fa0d25e","Type":"ContainerStarted","Data":"fdebea8d07f3ecab0e17d939389f609695219d32304ab7829a843a8633716bd9"}
Jun 02 15:46:46.712410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.712353    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kj5mf" event={"ID":"d9111172-9d4c-4285-9e39-ba0c9a637533","Type":"ContainerStarted","Data":"2d04dfcd27cc9447bf762cf5fcc3b50313c041cf957700cdda92a573a201d764"}
Jun 02 15:46:46.735799 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:46.735752    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-30.ec2.internal" podStartSLOduration=1.7357346040000001 podStartE2EDuration="1.735734604s" podCreationTimestamp="2026-06-02 15:46:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:46.71954211 +0000 UTC m=+3.595068272" watchObservedRunningTime="2026-06-02 15:46:46.735734604 +0000 UTC m=+3.611260764"
Jun 02 15:46:47.190875 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:47.190778    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:47.191047 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.191008    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:47.191104 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.191074    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:49.191054448 +0000 UTC m=+6.066580610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:47.291963 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:47.291922    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:47.292151 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.292134    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:47.292212 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.292154    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:47.292212 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.292167    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:47.292337 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.292226    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:46:49.292207166 +0000 UTC m=+6.167733322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:47.680555 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:47.680524    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:47.681011 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:47.680524    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:47.681011 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.680672    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:47.681011 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:47.680729    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:47.743481 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:47.743425    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal" event={"ID":"5c5b9144fc1c1a59765ef243be3273ec","Type":"ContainerStarted","Data":"62c8108fe5c58aca7cf7c01a522d16f3c5441440d395d1c52fc5f8c0451e9b4c"}
Jun 02 15:46:49.207117 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:49.207074    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:49.207608 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.207314    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:49.207608 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.207395    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:53.207360134 +0000 UTC m=+10.082886294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:49.308353 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:49.308314    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:49.308544 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.308504    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:49.308544 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.308528    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:49.308544 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.308541    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:49.308694 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.308600    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:46:53.308581063 +0000 UTC m=+10.184107217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:49.679533 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:49.679496    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:49.679704 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:49.679555    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:49.679759 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.679707    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:49.680167 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:49.680137    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:51.679562 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:51.679526    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:51.679966 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:51.679665    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:51.679966 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:51.679857    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:51.680039 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:51.679963    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:53.243310 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:53.243077    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:53.243803 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.243214    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:53.243803 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.243487    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:01.243470225 +0000 UTC m=+18.118996364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:53.344556 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:53.344463    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:53.344818 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.344667    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:53.344818 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.344694    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:53.344818 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.344708    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:53.344818 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.344769    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:47:01.344749248 +0000 UTC m=+18.220275403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:53.679852 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:53.679766    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:53.679997 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.679901    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:53.680230 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:53.680080    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:53.680230 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:53.680185    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:55.679763 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:55.679722    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:55.679763 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:55.679763    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:55.680272 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:55.679872    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:55.680272 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:55.680004    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:57.679622 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:57.679539    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:57.679622 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:57.679562    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:57.680106 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:57.679692    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:57.680106 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:57.679795    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:46:59.679412 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:59.679357    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:46:59.679412 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:46:59.679401    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:46:59.679932 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:59.679519    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:46:59.679932 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:46:59.679676    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:01.294671 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:01.294629    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:01.295137 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.294823    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:01.295137 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.294904    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:17.294882112 +0000 UTC m=+34.170408254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:01.395950 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:01.395902    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:01.396142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.396089    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:01.396142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.396111    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:01.396142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.396126    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:01.396290 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.396191    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:47:17.396172477 +0000 UTC m=+34.271698617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:01.679680 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:01.679596    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:01.679842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:01.679615    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:01.679842 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.679739    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:01.679924 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:01.679836    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:03.679926 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:03.679892    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:03.680495 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:03.679945    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:03.680495 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:03.680013    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:03.680495 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:03.680067    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:04.778443 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.778406    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjg4n" event={"ID":"6e1a921c-9997-4cd6-937f-a1a1a2967260","Type":"ContainerStarted","Data":"393648d4365e47534282eef7cd2bf35f86ebe9a8d24f641fe7c87bf740de510b"}
Jun 02 15:47:04.779953 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.779930    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r4wc4" event={"ID":"ea971506-fe66-418b-a884-855d031a66aa","Type":"ContainerStarted","Data":"ce8a505fce355e58bf5a26e13df285dced621a98441ced66c9c8ccf7b6340dc4"}
Jun 02 15:47:04.781268 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.781244    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="00c6448fceb0296ef98828f40e10bee85688542cc7dccc5afc4aa54eef7a772e" exitCode=0
Jun 02 15:47:04.781360 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.781299    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"00c6448fceb0296ef98828f40e10bee85688542cc7dccc5afc4aa54eef7a772e"}
Jun 02 15:47:04.783447 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.783423    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gbcbz" event={"ID":"51da022a-4320-403f-bee1-05113613afb4","Type":"ContainerStarted","Data":"66eeac05d204dc907bdf9a4809739a1707dead6c2d0fbcea0cd4b7e3ad728c32"}
Jun 02 15:47:04.784869 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.784852    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:47:04.785165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.785147    2571 generic.go:358] "Generic (PLEG): container finished" podID="631ad240-3383-480d-ab82-bde14902804e" containerID="95f0d9f4d0cb288a3cc86bf8dd348d5d01bec2ca2e29e49619163cc45b6e1bb7" exitCode=1
Jun 02 15:47:04.785227 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.785205    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerDied","Data":"95f0d9f4d0cb288a3cc86bf8dd348d5d01bec2ca2e29e49619163cc45b6e1bb7"}
Jun 02 15:47:04.785280 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.785228    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"be72a9fa46431790bf522b42883b38a3eeaebbb0bd97abc3aae8aff4ea4220d1"}
Jun 02 15:47:04.786526 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.786509    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh" event={"ID":"6edf4f75-5e8d-416a-b930-c64d36a81a02","Type":"ContainerStarted","Data":"3879c7c72f6387fca0099ac91aace92a93949d89b5ac969699524b4cd8ae6a67"}
Jun 02 15:47:04.789798 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.789774    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wbzp" event={"ID":"3e5af621-7c9a-437e-9195-e97b1fa0d25e","Type":"ContainerStarted","Data":"dc6a6c0dc4d7833bce8f05eaffa716147c7eefa811449400e9198cd81d99a9ef"}
Jun 02 15:47:04.791129 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.791107    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kj5mf" event={"ID":"d9111172-9d4c-4285-9e39-ba0c9a637533","Type":"ContainerStarted","Data":"150faf314a6c15af82682877c0214a4499c0dbf74aa34b0d61871235201ed021"}
Jun 02 15:47:04.793758 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.793721    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wjg4n" podStartSLOduration=3.733524521 podStartE2EDuration="21.793711091s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.268505283 +0000 UTC m=+3.144031436" lastFinishedPulling="2026-06-02 15:47:04.328691866 +0000 UTC m=+21.204218006" observedRunningTime="2026-06-02 15:47:04.793261793 +0000 UTC m=+21.668787955" watchObservedRunningTime="2026-06-02 15:47:04.793711091 +0000 UTC m=+21.669237252"
Jun 02 15:47:04.793842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.793795    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-30.ec2.internal" podStartSLOduration=19.793791326 podStartE2EDuration="19.793791326s" podCreationTimestamp="2026-06-02 15:46:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:47.761670837 +0000 UTC m=+4.637196999" watchObservedRunningTime="2026-06-02 15:47:04.793791326 +0000 UTC m=+21.669317487"
Jun 02 15:47:04.812311 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.812271    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-gbcbz" podStartSLOduration=3.715017187 podStartE2EDuration="21.812258065s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.264268868 +0000 UTC m=+3.139795009" lastFinishedPulling="2026-06-02 15:47:04.361509745 +0000 UTC m=+21.237035887" observedRunningTime="2026-06-02 15:47:04.812247981 +0000 UTC m=+21.687774143" watchObservedRunningTime="2026-06-02 15:47:04.812258065 +0000 UTC m=+21.687784204"
Jun 02 15:47:04.831012 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.830965    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kj5mf" podStartSLOduration=8.650210691 podStartE2EDuration="21.83095012s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.256479741 +0000 UTC m=+3.132005883" lastFinishedPulling="2026-06-02 15:46:59.437219158 +0000 UTC m=+16.312745312" observedRunningTime="2026-06-02 15:47:04.8306694 +0000 UTC m=+21.706195560" watchObservedRunningTime="2026-06-02 15:47:04.83095012 +0000 UTC m=+21.706476302"
Jun 02 15:47:04.848322 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.848279    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-r4wc4" podStartSLOduration=8.679442332 podStartE2EDuration="21.848262406s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.268397354 +0000 UTC m=+3.143923496" lastFinishedPulling="2026-06-02 15:46:59.43721742 +0000 UTC m=+16.312743570" observedRunningTime="2026-06-02 15:47:04.847518359 +0000 UTC m=+21.723044532" watchObservedRunningTime="2026-06-02 15:47:04.848262406 +0000 UTC m=+21.723788567"
Jun 02 15:47:04.870510 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:04.870471    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7wbzp" podStartSLOduration=3.77640864 podStartE2EDuration="21.870458024s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.267510048 +0000 UTC m=+3.143036190" lastFinishedPulling="2026-06-02 15:47:04.361559424 +0000 UTC m=+21.237085574" observedRunningTime="2026-06-02 15:47:04.866305471 +0000 UTC m=+21.741831631" watchObservedRunningTime="2026-06-02 15:47:04.870458024 +0000 UTC m=+21.745984184"
Jun 02 15:47:05.679421 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.679226    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:05.679566 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.679233    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:05.679632 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:05.679610    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:05.679679 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:05.679494    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:05.737191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.737157    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:47:05.794768 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.794702    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:47:05.795174 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.795051    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"737ffb88e827f86511e994bd3a074c16b47d2ef085784e50eb2d92c774bc1a1c"}
Jun 02 15:47:05.795174 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.795082    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"7827c6d2bc1dad9a9a2e158d161194890968c2a609a8fcfe14c0040ff2250e7b"}
Jun 02 15:47:05.795174 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.795091    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"0893066f81828e27729bbcb7ab78cd38d70ac6eccd418a83ed09d7cabb0636d3"}
Jun 02 15:47:05.795174 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.795100    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"e72c2c0428023efe78ccbfb0e837e8dba3ad37861c46fea3415de2181fd09dd9"}
Jun 02 15:47:05.796535 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:05.796501    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh" event={"ID":"6edf4f75-5e8d-416a-b930-c64d36a81a02","Type":"ContainerStarted","Data":"bc79fec077cf56bb026ff26f81e1813da9db1935fd7a8df231836c0342c6863c"}
Jun 02 15:47:06.634879 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.634771    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:47:05.737183162Z","UUID":"33886b05-50a1-47bf-a2a3-bb95a2f7ca86","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:47:06.638242 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.638220    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:47:06.638242 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.638247    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:47:06.799533 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.799496    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5lv9" event={"ID":"3b4a915b-6de0-4eac-8cf7-a4d48060b2e4","Type":"ContainerStarted","Data":"61e7e6ab81ae18eac7c65902b8a68c0e6d06b64c5fbac93c4ceaae5d3d0b472f"}
Jun 02 15:47:06.801544 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.801509    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh" event={"ID":"6edf4f75-5e8d-416a-b930-c64d36a81a02","Type":"ContainerStarted","Data":"6b6860f7c983baa43916a952b5753b3b42058ec90402a748764239382bb1418b"}
Jun 02 15:47:06.815173 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.815129    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b5lv9" podStartSLOduration=10.637396285 podStartE2EDuration="23.815111344s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.25950536 +0000 UTC m=+3.135031500" lastFinishedPulling="2026-06-02 15:46:59.437220406 +0000 UTC m=+16.312746559" observedRunningTime="2026-06-02 15:47:06.814396896 +0000 UTC m=+23.689923058" watchObservedRunningTime="2026-06-02 15:47:06.815111344 +0000 UTC m=+23.690637507"
Jun 02 15:47:06.832380 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:06.832319    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clkrh" podStartSLOduration=3.441747894 podStartE2EDuration="23.832304506s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.257676691 +0000 UTC m=+3.133202834" lastFinishedPulling="2026-06-02 15:47:06.648233302 +0000 UTC m=+23.523759446" observedRunningTime="2026-06-02 15:47:06.832240281 +0000 UTC m=+23.707766441" watchObservedRunningTime="2026-06-02 15:47:06.832304506 +0000 UTC m=+23.707830668"
Jun 02 15:47:07.679904 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.679865    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:07.680083 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.679865    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:07.680083 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:07.680009    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:07.680159 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:07.680113    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:07.802581 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.802542    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:47:07.803335 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.803314    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:47:07.806686 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.806665    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:47:07.807193 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:07.807121    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"ee08ac6259d0ff8ac6193952fde02913fe4895e4e894de162d84a53f09c4a9f8"}
Jun 02 15:47:08.809304 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:08.809268    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:47:08.810289 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:08.810252    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kj5mf"
Jun 02 15:47:09.679819 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.679524    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:09.679974 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.679573    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:09.679974 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:09.679882    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:09.679974 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:09.679930    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:09.812720 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.812687    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="0ac87242f053bc94820a366fa4f71278aab9e829e109c423b46173a48ceab143" exitCode=0
Jun 02 15:47:09.813217 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.812767    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"0ac87242f053bc94820a366fa4f71278aab9e829e109c423b46173a48ceab143"}
Jun 02 15:47:09.815850 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.815828    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:47:09.816265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.816239    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"d29303486437d4502dbc7a10e7c244bff8aa90f0e9e691cce1125174f3a31c6e"}
Jun 02 15:47:09.816646 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.816630    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:09.816762 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.816656    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:09.816762 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.816668    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:09.816871 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.816772    2571 scope.go:117] "RemoveContainer" containerID="95f0d9f4d0cb288a3cc86bf8dd348d5d01bec2ca2e29e49619163cc45b6e1bb7"
Jun 02 15:47:09.832885 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.832859    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:09.832993 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:09.832930    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:10.822323 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:10.822298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:47:10.822730 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:10.822683    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" event={"ID":"631ad240-3383-480d-ab82-bde14902804e","Type":"ContainerStarted","Data":"2cafa87f8d3e865453429a0680465f7b7c4dd7458cc46eca686c8dd712792526"}
Jun 02 15:47:10.854527 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:10.854473    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m" podStartSLOduration=9.684531634 podStartE2EDuration="27.854459022s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.258868941 +0000 UTC m=+3.134395094" lastFinishedPulling="2026-06-02 15:47:04.428796344 +0000 UTC m=+21.304322482" observedRunningTime="2026-06-02 15:47:10.853541322 +0000 UTC m=+27.729067517" watchObservedRunningTime="2026-06-02 15:47:10.854459022 +0000 UTC m=+27.729985183"
Jun 02 15:47:11.325275 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.325056    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmkbn"]
Jun 02 15:47:11.325429 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.325321    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:11.325479 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:11.325421    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:11.328842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.328819    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qqr9"]
Jun 02 15:47:11.328949 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.328919    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:11.329015 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:11.328998    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:11.826490 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.826450    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="079450ebebfe6d060f78bb916a6f7dbbad0298db9e4fa49329cc141c5c76bf13" exitCode=0
Jun 02 15:47:11.826490 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:11.826477    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"079450ebebfe6d060f78bb916a6f7dbbad0298db9e4fa49329cc141c5c76bf13"}
Jun 02 15:47:12.679546 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:12.679505    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:12.679743 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:12.679518    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:12.679743 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:12.679612    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:12.679743 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:12.679698    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:13.832164 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:13.832125    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="6551daa8ecb225130c6fc0f592c2436f37dc078d98e47ea42075183207812d3e" exitCode=0
Jun 02 15:47:13.832701 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:13.832200    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"6551daa8ecb225130c6fc0f592c2436f37dc078d98e47ea42075183207812d3e"}
Jun 02 15:47:14.679669 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:14.679631    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:14.679852 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:14.679631    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:14.679852 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:14.679788    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:14.679852 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:14.679823    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:16.679592 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:16.679355    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:16.680017 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:16.679358    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:16.680017 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:16.679693    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmkbn" podUID="68584f01-5fd0-42c0-a157-392408a5f2cb"
Jun 02 15:47:16.680017 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:16.679748    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:47:17.313099 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:17.313050    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:17.313297 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.313230    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:17.313359 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.313306    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:49.313291279 +0000 UTC m=+66.188817421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:17.413880 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:17.413847    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:17.414058 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.414006    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:17.414058 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.414026    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:17.414058 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.414036    2571 projected.go:194] Error preparing data for projected volume kube-api-access-28bvb for pod openshift-network-diagnostics/network-check-target-wmkbn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:17.414170 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:17.414086    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb podName:68584f01-5fd0-42c0-a157-392408a5f2cb nodeName:}" failed. No retries permitted until 2026-06-02 15:47:49.414072968 +0000 UTC m=+66.289599113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-28bvb" (UniqueName: "kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb") pod "network-check-target-wmkbn" (UID: "68584f01-5fd0-42c0-a157-392408a5f2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:18.491250 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.491219    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-131-30.ec2.internal" event="NodeReady"
Jun 02 15:47:18.491777 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.491386    2571 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:47:18.590744 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.590713    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v22gh"]
Jun 02 15:47:18.603914 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.603882    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5wfsn"]
Jun 02 15:47:18.604077 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.604054    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.609725 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.609692    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ht87l\""
Jun 02 15:47:18.609725 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.609691    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:47:18.610397 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.610354    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:47:18.620356 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.620332    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v22gh"]
Jun 02 15:47:18.620511 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.620485    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:18.624609 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.624580    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5wfsn"]
Jun 02 15:47:18.629450 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.629414    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:47:18.629718 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.629701    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cw7rp\""
Jun 02 15:47:18.629827 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.629741    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:47:18.629827 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.629773    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:47:18.679082 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.679054    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:18.679271 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.679095    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:18.682580 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.682407    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:18.682580 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.682439    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mqvtg\""
Jun 02 15:47:18.682785 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.682692    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq5xh\""
Jun 02 15:47:18.682785 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.682724    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:18.682891 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.682798    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:18.724120 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.724287 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllt2\" (UniqueName: \"kubernetes.io/projected/b0d74645-7b90-430f-af77-f554a9418180-kube-api-access-bllt2\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.724287 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvlj\" (UniqueName: \"kubernetes.io/projected/e3b6c156-e1a0-4290-9a47-55dd7016276e-kube-api-access-hgvlj\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:18.724287 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d74645-7b90-430f-af77-f554a9418180-tmp-dir\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.724478 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724325    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d74645-7b90-430f-af77-f554a9418180-config-volume\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.724478 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.724359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:18.825597 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825506    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.825597 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825570    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bllt2\" (UniqueName: \"kubernetes.io/projected/b0d74645-7b90-430f-af77-f554a9418180-kube-api-access-bllt2\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.825814 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825604    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvlj\" (UniqueName: \"kubernetes.io/projected/e3b6c156-e1a0-4290-9a47-55dd7016276e-kube-api-access-hgvlj\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:18.825814 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825629    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d74645-7b90-430f-af77-f554a9418180-tmp-dir\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.825814 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:18.825670    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:18.825814 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:18.825734    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:19.325714576 +0000 UTC m=+36.201240724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:18.826022 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825854    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d74645-7b90-430f-af77-f554a9418180-config-volume\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.826022 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.825916    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:18.826140 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:18.826026    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:18.826140 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:18.826078    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:19.326062527 +0000 UTC m=+36.201588673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:18.826236 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.826161    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d74645-7b90-430f-af77-f554a9418180-tmp-dir\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.837968 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.837933    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d74645-7b90-430f-af77-f554a9418180-config-volume\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.852936 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.852907    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllt2\" (UniqueName: \"kubernetes.io/projected/b0d74645-7b90-430f-af77-f554a9418180-kube-api-access-bllt2\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:18.853155 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:18.853125    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvlj\" (UniqueName: \"kubernetes.io/projected/e3b6c156-e1a0-4290-9a47-55dd7016276e-kube-api-access-hgvlj\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:19.330001 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:19.329959    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:19.330172 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:19.330031    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:19.330172 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:19.330130    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:19.330262 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:19.330177    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:19.330262 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:19.330192    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:20.330175261 +0000 UTC m=+37.205701399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:19.330262 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:19.330218    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:20.330205773 +0000 UTC m=+37.205731912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:20.337962 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:20.337923    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:20.338334 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:20.338065    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:20.338334 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:20.338090    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:20.338334 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:20.338125    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:22.338109165 +0000 UTC m=+39.213635304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:20.338334 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:20.338167    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:20.338334 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:20.338216    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:22.338205173 +0000 UTC m=+39.213731311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:20.849027 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:20.848992    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="c342f75b72e3cf87e4de7a1b55dedbe44b74e62d26d1e60a1f5aab4722d440c0" exitCode=0
Jun 02 15:47:20.849188 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:20.849057    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"c342f75b72e3cf87e4de7a1b55dedbe44b74e62d26d1e60a1f5aab4722d440c0"}
Jun 02 15:47:21.852830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:21.852794    2571 generic.go:358] "Generic (PLEG): container finished" podID="a71f5380-9081-4609-8da0-42da280a618d" containerID="9bc249bb64a05a0aa6af73fcc4598b26143e378fd4e334696d1cc7ce6e4e5b21" exitCode=0
Jun 02 15:47:21.853309 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:21.852850    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerDied","Data":"9bc249bb64a05a0aa6af73fcc4598b26143e378fd4e334696d1cc7ce6e4e5b21"}
Jun 02 15:47:22.353153 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:22.352973    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:22.353329 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:22.353165    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:22.353329 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:22.353117    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:22.353329 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:22.353251    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:22.353329 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:22.353278    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:26.353257061 +0000 UTC m=+43.228783207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:22.353329 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:22.353297    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:26.353288749 +0000 UTC m=+43.228814888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:22.857575 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:22.857545    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" event={"ID":"a71f5380-9081-4609-8da0-42da280a618d","Type":"ContainerStarted","Data":"42e0da576c82430e58cc9097f3db013ca05fccc2d43674e64b0b6e4c7b30e8d2"}
Jun 02 15:47:22.890319 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:22.890270    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xzf5d" podStartSLOduration=6.238162339 podStartE2EDuration="39.890255549s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:46:46.266822084 +0000 UTC m=+3.142348238" lastFinishedPulling="2026-06-02 15:47:19.918915293 +0000 UTC m=+36.794441448" observedRunningTime="2026-06-02 15:47:22.887469129 +0000 UTC m=+39.762995289" watchObservedRunningTime="2026-06-02 15:47:22.890255549 +0000 UTC m=+39.765781749"
Jun 02 15:47:26.377298 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:26.377256    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:26.377795 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:26.377336    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:26.377795 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:26.377411    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:26.377795 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:26.377462    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:26.377795 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:26.377472    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:34.377456427 +0000 UTC m=+51.252982566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:26.377795 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:26.377511    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:34.377494826 +0000 UTC m=+51.253020990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:34.427671 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:34.427631    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:34.428144 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:34.427712    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:34.428144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:34.427783    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:34.428144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:34.427836    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:50.427823143 +0000 UTC m=+67.303349283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:34.428144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:34.427783    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:34.428144 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:34.427899    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:50.427887664 +0000 UTC m=+67.303413819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:41.837062 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:41.837030    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9st6m"
Jun 02 15:47:49.335462 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.335411    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:47:49.338052 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.338034    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:49.346569 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:49.346543    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:47:49.346659 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:49.346621    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:53.346598491 +0000 UTC m=+130.222124644 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : secret "metrics-daemon-secret" not found
Jun 02 15:47:49.436095 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.436054    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:49.438963 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.438941    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:49.449566 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.449537    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:49.461012 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.460990    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bvb\" (UniqueName: \"kubernetes.io/projected/68584f01-5fd0-42c0-a157-392408a5f2cb-kube-api-access-28bvb\") pod \"network-check-target-wmkbn\" (UID: \"68584f01-5fd0-42c0-a157-392408a5f2cb\") " pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:49.600293 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.600213    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gq5xh\""
Jun 02 15:47:49.607445 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.607419    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:49.760961 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.760926    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmkbn"]
Jun 02 15:47:49.765626 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:47:49.765602    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68584f01_5fd0_42c0_a157_392408a5f2cb.slice/crio-17ff1be1a9647537410d260044cf7f7e3040a61e3124268b759e1acb5cacacb8 WatchSource:0}: Error finding container 17ff1be1a9647537410d260044cf7f7e3040a61e3124268b759e1acb5cacacb8: Status 404 returned error can't find the container with id 17ff1be1a9647537410d260044cf7f7e3040a61e3124268b759e1acb5cacacb8
Jun 02 15:47:49.906260 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:49.906176    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmkbn" event={"ID":"68584f01-5fd0-42c0-a157-392408a5f2cb","Type":"ContainerStarted","Data":"17ff1be1a9647537410d260044cf7f7e3040a61e3124268b759e1acb5cacacb8"}
Jun 02 15:47:50.443482 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:50.443446    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:47:50.443947 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:50.443520    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:47:50.443947 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:50.443605    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:50.443947 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:50.443676    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:22.443660841 +0000 UTC m=+99.319186984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:50.443947 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:50.443605    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:50.443947 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:47:50.443718    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:48:22.443709176 +0000 UTC m=+99.319235315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:47:52.913564 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:52.913521    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmkbn" event={"ID":"68584f01-5fd0-42c0-a157-392408a5f2cb","Type":"ContainerStarted","Data":"d8e6a843925757e3dd3bb2e539e9e8d16f3b40a46985a8f970ae18da3843ed04"}
Jun 02 15:47:52.914031 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:52.913645    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:47:52.951258 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:47:52.951204    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wmkbn" podStartSLOduration=67.33243599 podStartE2EDuration="1m9.951188557s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:47:49.7679736 +0000 UTC m=+66.643499742" lastFinishedPulling="2026-06-02 15:47:52.386726167 +0000 UTC m=+69.262252309" observedRunningTime="2026-06-02 15:47:52.950612989 +0000 UTC m=+69.826139151" watchObservedRunningTime="2026-06-02 15:47:52.951188557 +0000 UTC m=+69.826714772"
Jun 02 15:48:22.457785 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:22.457745    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:48:22.458192 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:22.457799    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:48:22.458192 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:22.457895    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:48:22.458192 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:22.457897    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:48:22.458192 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:22.457957    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls podName:b0d74645-7b90-430f-af77-f554a9418180 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:26.457943289 +0000 UTC m=+163.333469428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls") pod "dns-default-v22gh" (UID: "b0d74645-7b90-430f-af77-f554a9418180") : secret "dns-default-metrics-tls" not found
Jun 02 15:48:22.458192 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:22.457970    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert podName:e3b6c156-e1a0-4290-9a47-55dd7016276e nodeName:}" failed. No retries permitted until 2026-06-02 15:49:26.457964314 +0000 UTC m=+163.333490453 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert") pod "ingress-canary-5wfsn" (UID: "e3b6c156-e1a0-4290-9a47-55dd7016276e") : secret "canary-serving-cert" not found
Jun 02 15:48:23.917775 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:23.917747    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wmkbn"
Jun 02 15:48:39.338201 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.338165    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-788bf58df5-c2bl2"]
Jun 02 15:48:39.340934 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.340916    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.343218 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.343191    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bc549f7b9-ncs76"]
Jun 02 15:48:39.344947 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.344916    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jun 02 15:48:39.344947 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.344927    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:48:39.344947 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.344925    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.345456 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.345440    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ghz5c\""
Jun 02 15:48:39.345900 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.345880    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:48:39.345900 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.345894    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jun 02 15:48:39.351960 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.351934    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jun 02 15:48:39.352070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.351985    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jun 02 15:48:39.352070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.352009    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jun 02 15:48:39.352181 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.351941    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jun 02 15:48:39.352181 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.352172    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jun 02 15:48:39.352286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.352183    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x8bn8\""
Jun 02 15:48:39.352336 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.352290    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jun 02 15:48:39.355360 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.355322    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jun 02 15:48:39.367179 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.367155    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-788bf58df5-c2bl2"]
Jun 02 15:48:39.374297 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374274    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bc549f7b9-ncs76"]
Jun 02 15:48:39.374537 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.374590 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-default-certificate\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.374590 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-service-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.374654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.374690 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsv4\" (UniqueName: \"kubernetes.io/projected/34318a4f-dd28-449c-ae48-84eec2db293b-kube-api-access-xxsv4\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.374722 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-stats-auth\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.374722 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-tmp\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.374793 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.374793 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2db81b66-10f3-4fc9-a130-484b26440831-serving-cert\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.374857 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-snapshots\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.374857 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.374822    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8nt\" (UniqueName: \"kubernetes.io/projected/2db81b66-10f3-4fc9-a130-484b26440831-kube-api-access-4f8nt\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.447475 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.447439    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"]
Jun 02 15:48:39.449436 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.449414    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.457562 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.457543    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:48:39.457908 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.457892    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-946qt\""
Jun 02 15:48:39.462170 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.462156    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:48:39.462477 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.462460    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jun 02 15:48:39.462519 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.462498    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jun 02 15:48:39.469698 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.469678    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"]
Jun 02 15:48:39.476059 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476038    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476067    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-default-certificate\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.476165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.476165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476116    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-service-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476142    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.476165 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476157    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsv4\" (UniqueName: \"kubernetes.io/projected/34318a4f-dd28-449c-ae48-84eec2db293b-kube-api-access-xxsv4\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476176    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-stats-auth\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476192    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-tmp\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476210    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476236    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2db81b66-10f3-4fc9-a130-484b26440831-serving-cert\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-telemetry-config\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476293    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-snapshots\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476318    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8nt\" (UniqueName: \"kubernetes.io/projected/2db81b66-10f3-4fc9-a130-484b26440831-kube-api-access-4f8nt\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476410 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.476332    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:39.976311974 +0000 UTC m=+116.851838137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:39.476819 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476422    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6plw\" (UniqueName: \"kubernetes.io/projected/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-kube-api-access-s6plw\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.476819 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.476690    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:39.476819 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476756    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-service-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.476819 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.476774    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:39.976755705 +0000 UTC m=+116.852281848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:39.476970 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.476944    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2db81b66-10f3-4fc9-a130-484b26440831-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.477264 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.477239    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-snapshots\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.477531 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.477509    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db81b66-10f3-4fc9-a130-484b26440831-tmp\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.478746 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.478718    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2db81b66-10f3-4fc9-a130-484b26440831-serving-cert\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.478822 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.478743    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-stats-auth\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.478822 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.478799    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-default-certificate\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.498888 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.498859    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8nt\" (UniqueName: \"kubernetes.io/projected/2db81b66-10f3-4fc9-a130-484b26440831-kube-api-access-4f8nt\") pod \"insights-operator-788bf58df5-c2bl2\" (UID: \"2db81b66-10f3-4fc9-a130-484b26440831\") " pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.510196 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.510173    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsv4\" (UniqueName: \"kubernetes.io/projected/34318a4f-dd28-449c-ae48-84eec2db293b-kube-api-access-xxsv4\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.577578 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.577539    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-telemetry-config\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.577771 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.577614    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s6plw\" (UniqueName: \"kubernetes.io/projected/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-kube-api-access-s6plw\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.577771 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.577709    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.577870 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.577832    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:39.577910 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.577905    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:48:40.077888372 +0000 UTC m=+116.953414518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:39.578288 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.578270    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-telemetry-config\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.594666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.594600    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6plw\" (UniqueName: \"kubernetes.io/projected/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-kube-api-access-s6plw\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:39.653898 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.653842    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-788bf58df5-c2bl2"
Jun 02 15:48:39.803065 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.803022    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-788bf58df5-c2bl2"]
Jun 02 15:48:39.806828 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:48:39.806798    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db81b66_10f3_4fc9_a130_484b26440831.slice/crio-b365aea074d4ec27994804e541c9ee6ba87605189e83b27b139fa1f91614299e WatchSource:0}: Error finding container b365aea074d4ec27994804e541c9ee6ba87605189e83b27b139fa1f91614299e: Status 404 returned error can't find the container with id b365aea074d4ec27994804e541c9ee6ba87605189e83b27b139fa1f91614299e
Jun 02 15:48:39.981950 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.981916    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.982134 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.982010    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:39.982134 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.982089    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:39.982134 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.982115    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:40.982102029 +0000 UTC m=+117.857628169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:39.982267 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:39.982153    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:40.982135243 +0000 UTC m=+117.857661399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:39.999654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:39.999614    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-c2bl2" event={"ID":"2db81b66-10f3-4fc9-a130-484b26440831","Type":"ContainerStarted","Data":"b365aea074d4ec27994804e541c9ee6ba87605189e83b27b139fa1f91614299e"}
Jun 02 15:48:40.082348 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:40.082309    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:40.082538 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:40.082473    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:40.082576 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:40.082539    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:48:41.082523945 +0000 UTC m=+117.958050083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:40.989597 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:40.989540    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:40.989965 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:40.989690    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:40.989965 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:40.989698    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:40.989965 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:40.989782    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:42.989766622 +0000 UTC m=+119.865292761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:40.989965 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:40.989796    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:42.989790665 +0000 UTC m=+119.865316804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:41.090208 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:41.090173    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:41.090401 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:41.090350    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:41.090462 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:41.090445    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:48:43.090423531 +0000 UTC m=+119.965949670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:42.005216 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:42.005178    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-c2bl2" event={"ID":"2db81b66-10f3-4fc9-a130-484b26440831","Type":"ContainerStarted","Data":"028a8ec42a9d3fde6ef179973e6ed356e3f6eeea83306158e662e9d9e9fed7d5"}
Jun 02 15:48:43.007159 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:43.007126    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:43.007579 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:43.007174    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:43.007579 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:43.007290    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:47.007274984 +0000 UTC m=+123.882801124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:43.007579 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:43.007326    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:43.007579 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:43.007399    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:47.007385505 +0000 UTC m=+123.882911643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:43.107780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:43.107746    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:43.107972 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:43.107896    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:43.107972 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:43.107969    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:48:47.107950752 +0000 UTC m=+123.983476909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:45.083141 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:45.083111    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r4wc4_ea971506-fe66-418b-a884-855d031a66aa/dns-node-resolver/0.log"
Jun 02 15:48:46.058467 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:46.058442    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjg4n_6e1a921c-9997-4cd6-937f-a1a1a2967260/node-ca/0.log"
Jun 02 15:48:47.039342 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:47.039313    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:47.039342 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:47.039349    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:47.039771 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:47.039480    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:47.039771 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:47.039506    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:55.039486791 +0000 UTC m=+131.915012945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:47.039771 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:47.039526    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:48:55.039519404 +0000 UTC m=+131.915045546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:47.139895 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:47.139863    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:47.140066 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:47.140008    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:47.140105 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:47.140080    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:48:55.140064503 +0000 UTC m=+132.015590641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:49.305507 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.305450    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-788bf58df5-c2bl2" podStartSLOduration=8.467708589 podStartE2EDuration="10.305432654s" podCreationTimestamp="2026-06-02 15:48:39 +0000 UTC" firstStartedPulling="2026-06-02 15:48:39.808505675 +0000 UTC m=+116.684031814" lastFinishedPulling="2026-06-02 15:48:41.646229737 +0000 UTC m=+118.521755879" observedRunningTime="2026-06-02 15:48:42.028188481 +0000 UTC m=+118.903714643" watchObservedRunningTime="2026-06-02 15:48:49.305432654 +0000 UTC m=+126.180958814"
Jun 02 15:48:49.305895 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.305774    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"]
Jun 02 15:48:49.308568 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.308554    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.310966 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.310946    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jun 02 15:48:49.311085 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.310985    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lhrch\""
Jun 02 15:48:49.311208 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.311196    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.311818 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.311802    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.311866 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.311850    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jun 02 15:48:49.318195 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.318173    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"]
Jun 02 15:48:49.358236 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.358202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f118d48-4f19-4dc1-a22a-b456f271c48c-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.358409 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.358313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f118d48-4f19-4dc1-a22a-b456f271c48c-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.358409 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.358350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6np\" (UniqueName: \"kubernetes.io/projected/8f118d48-4f19-4dc1-a22a-b456f271c48c-kube-api-access-zc6np\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.403049 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.403015    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"]
Jun 02 15:48:49.405887 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.405872    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.408634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.408609    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.408790 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.408753    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-thw4j\""
Jun 02 15:48:49.409251 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.408848    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.409251 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.409057    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jun 02 15:48:49.410245 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.410229    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"]
Jun 02 15:48:49.412830 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.412811    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.414683 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.414648    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"]
Jun 02 15:48:49.415292 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.415277    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jun 02 15:48:49.415445 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.415428    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.415613 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.415596    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jun 02 15:48:49.415655 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.415637    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pf4zv\""
Jun 02 15:48:49.415862 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.415844    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.417803 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.417786    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"
Jun 02 15:48:49.420044 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.420025    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.420662 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.420645    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dx2dc\""
Jun 02 15:48:49.423127 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.423108    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.440306 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.440275    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"]
Jun 02 15:48:49.441037 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.441017    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"]
Jun 02 15:48:49.441711 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.441692    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"]
Jun 02 15:48:49.458917 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.458889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b97419-e9e4-4f02-b246-80248eeb0025-config\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.459055 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.458923    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp4k\" (UniqueName: \"kubernetes.io/projected/36b97419-e9e4-4f02-b246-80248eeb0025-kube-api-access-hzp4k\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.459055 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.458947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b97419-e9e4-4f02-b246-80248eeb0025-serving-cert\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.459055 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459042    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f118d48-4f19-4dc1-a22a-b456f271c48c-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.459210 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znf4n\" (UniqueName: \"kubernetes.io/projected/6fa6ba27-492c-4831-be65-53e14d4d825d-kube-api-access-znf4n\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.459210 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459146    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6np\" (UniqueName: \"kubernetes.io/projected/8f118d48-4f19-4dc1-a22a-b456f271c48c-kube-api-access-zc6np\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.459306 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459277    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f118d48-4f19-4dc1-a22a-b456f271c48c-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.459361 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459329    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.459640 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.459623    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f118d48-4f19-4dc1-a22a-b456f271c48c-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.461362 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.461345    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f118d48-4f19-4dc1-a22a-b456f271c48c-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.475909 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.475881    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6np\" (UniqueName: \"kubernetes.io/projected/8f118d48-4f19-4dc1-a22a-b456f271c48c-kube-api-access-zc6np\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn2bq\" (UID: \"8f118d48-4f19-4dc1-a22a-b456f271c48c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.559979 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.559889    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.559979 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.559948    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b97419-e9e4-4f02-b246-80248eeb0025-config\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.559979 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.559967    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp4k\" (UniqueName: \"kubernetes.io/projected/36b97419-e9e4-4f02-b246-80248eeb0025-kube-api-access-hzp4k\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.560252 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.559987    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b97419-e9e4-4f02-b246-80248eeb0025-serving-cert\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.560252 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.560014    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-znf4n\" (UniqueName: \"kubernetes.io/projected/6fa6ba27-492c-4831-be65-53e14d4d825d-kube-api-access-znf4n\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.560252 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:49.560033    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:49.560252 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.560036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92x4\" (UniqueName: \"kubernetes.io/projected/299c6efd-0128-40ba-a067-fc0a3f51e7a6-kube-api-access-n92x4\") pod \"volume-data-source-validator-7598fd5bd-mxr8d\" (UID: \"299c6efd-0128-40ba-a067-fc0a3f51e7a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"
Jun 02 15:48:49.560252 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:49.560101    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls podName:6fa6ba27-492c-4831-be65-53e14d4d825d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:50.060081859 +0000 UTC m=+126.935607998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-wzg84" (UID: "6fa6ba27-492c-4831-be65-53e14d4d825d") : secret "samples-operator-tls" not found
Jun 02 15:48:49.560610 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.560593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b97419-e9e4-4f02-b246-80248eeb0025-config\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.562932 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.562912    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b97419-e9e4-4f02-b246-80248eeb0025-serving-cert\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.572789 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.572764    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp4k\" (UniqueName: \"kubernetes.io/projected/36b97419-e9e4-4f02-b246-80248eeb0025-kube-api-access-hzp4k\") pod \"service-ca-operator-69877444c-7wsjw\" (UID: \"36b97419-e9e4-4f02-b246-80248eeb0025\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.572907 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.572868    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znf4n\" (UniqueName: \"kubernetes.io/projected/6fa6ba27-492c-4831-be65-53e14d4d825d-kube-api-access-znf4n\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:49.617698 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.617669    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"
Jun 02 15:48:49.661402 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.661036    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n92x4\" (UniqueName: \"kubernetes.io/projected/299c6efd-0128-40ba-a067-fc0a3f51e7a6-kube-api-access-n92x4\") pod \"volume-data-source-validator-7598fd5bd-mxr8d\" (UID: \"299c6efd-0128-40ba-a067-fc0a3f51e7a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"
Jun 02 15:48:49.677895 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.677860    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92x4\" (UniqueName: \"kubernetes.io/projected/299c6efd-0128-40ba-a067-fc0a3f51e7a6-kube-api-access-n92x4\") pod \"volume-data-source-validator-7598fd5bd-mxr8d\" (UID: \"299c6efd-0128-40ba-a067-fc0a3f51e7a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"
Jun 02 15:48:49.722220 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.722177    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"
Jun 02 15:48:49.727909 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.727877    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"
Jun 02 15:48:49.737972 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.737945    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq"]
Jun 02 15:48:49.741736 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:48:49.741706    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f118d48_4f19_4dc1_a22a_b456f271c48c.slice/crio-6410a99716ceb7fbfbe4fc849e671cd8408eeeb1e5562051e51dd0f8342d64f8 WatchSource:0}: Error finding container 6410a99716ceb7fbfbe4fc849e671cd8408eeeb1e5562051e51dd0f8342d64f8: Status 404 returned error can't find the container with id 6410a99716ceb7fbfbe4fc849e671cd8408eeeb1e5562051e51dd0f8342d64f8
Jun 02 15:48:49.862782 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.862733    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw"]
Jun 02 15:48:49.865898 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:48:49.865869    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b97419_e9e4_4f02_b246_80248eeb0025.slice/crio-72c1d4d80462880623b1ebd9daf023a3923cf1e94212bc9a7249db9c4940c969 WatchSource:0}: Error finding container 72c1d4d80462880623b1ebd9daf023a3923cf1e94212bc9a7249db9c4940c969: Status 404 returned error can't find the container with id 72c1d4d80462880623b1ebd9daf023a3923cf1e94212bc9a7249db9c4940c969
Jun 02 15:48:49.877488 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:49.877460    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d"]
Jun 02 15:48:49.880257 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:48:49.880232    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299c6efd_0128_40ba_a067_fc0a3f51e7a6.slice/crio-caf06881212745e7cd28d4b85532b99c734835e1ee04f0ba74ae855ceb62b210 WatchSource:0}: Error finding container caf06881212745e7cd28d4b85532b99c734835e1ee04f0ba74ae855ceb62b210: Status 404 returned error can't find the container with id caf06881212745e7cd28d4b85532b99c734835e1ee04f0ba74ae855ceb62b210
Jun 02 15:48:50.020935 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:50.020897    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d" event={"ID":"299c6efd-0128-40ba-a067-fc0a3f51e7a6","Type":"ContainerStarted","Data":"caf06881212745e7cd28d4b85532b99c734835e1ee04f0ba74ae855ceb62b210"}
Jun 02 15:48:50.021869 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:50.021845    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw" event={"ID":"36b97419-e9e4-4f02-b246-80248eeb0025","Type":"ContainerStarted","Data":"72c1d4d80462880623b1ebd9daf023a3923cf1e94212bc9a7249db9c4940c969"}
Jun 02 15:48:50.022789 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:50.022760    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq" event={"ID":"8f118d48-4f19-4dc1-a22a-b456f271c48c","Type":"ContainerStarted","Data":"6410a99716ceb7fbfbe4fc849e671cd8408eeeb1e5562051e51dd0f8342d64f8"}
Jun 02 15:48:50.064456 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:50.064429    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:50.064602 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:50.064583    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:50.064657 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:50.064647    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls podName:6fa6ba27-492c-4831-be65-53e14d4d825d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:51.064632911 +0000 UTC m=+127.940159049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-wzg84" (UID: "6fa6ba27-492c-4831-be65-53e14d4d825d") : secret "samples-operator-tls" not found
Jun 02 15:48:51.073826 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:51.073776    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:51.074346 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:51.073953    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:51.074346 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:51.074033    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls podName:6fa6ba27-492c-4831-be65-53e14d4d825d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:53.074012041 +0000 UTC m=+129.949538203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-wzg84" (UID: "6fa6ba27-492c-4831-be65-53e14d4d825d") : secret "samples-operator-tls" not found
Jun 02 15:48:53.031283 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.031243    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d" event={"ID":"299c6efd-0128-40ba-a067-fc0a3f51e7a6","Type":"ContainerStarted","Data":"e1928c4af6bc83ee84c4dd7c6fea0d4ae36a8849021957f632c46331b365fb9b"}
Jun 02 15:48:53.032469 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.032436    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw" event={"ID":"36b97419-e9e4-4f02-b246-80248eeb0025","Type":"ContainerStarted","Data":"6abf3f7a6063bbbc43fe1e7b05f5ea1383a27e0e928df4c59f892905d58a8e92"}
Jun 02 15:48:53.033815 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.033792    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq" event={"ID":"8f118d48-4f19-4dc1-a22a-b456f271c48c","Type":"ContainerStarted","Data":"6d96d8a79ef28529027eaf9c2acb67a1cf28caaa2b84b5eb867be6ec33ec0ba6"}
Jun 02 15:48:53.050275 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.050222    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-7598fd5bd-mxr8d" podStartSLOduration=1.66018825 podStartE2EDuration="4.050207888s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:49.882054359 +0000 UTC m=+126.757580497" lastFinishedPulling="2026-06-02 15:48:52.272073986 +0000 UTC m=+129.147600135" observedRunningTime="2026-06-02 15:48:53.049801898 +0000 UTC m=+129.925328110" watchObservedRunningTime="2026-06-02 15:48:53.050207888 +0000 UTC m=+129.925734049"
Jun 02 15:48:53.066571 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.066527    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw" podStartSLOduration=1.658539501 podStartE2EDuration="4.066513407s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:49.867737331 +0000 UTC m=+126.743263469" lastFinishedPulling="2026-06-02 15:48:52.275711237 +0000 UTC m=+129.151237375" observedRunningTime="2026-06-02 15:48:53.065850175 +0000 UTC m=+129.941376337" watchObservedRunningTime="2026-06-02 15:48:53.066513407 +0000 UTC m=+129.942039568"
Jun 02 15:48:53.091027 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.090996    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:53.092747 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:53.092719    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:53.092872 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:53.092814    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls podName:6fa6ba27-492c-4831-be65-53e14d4d825d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:57.092794729 +0000 UTC m=+133.968320873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-wzg84" (UID: "6fa6ba27-492c-4831-be65-53e14d4d825d") : secret "samples-operator-tls" not found
Jun 02 15:48:53.093982 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.093933    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq" podStartSLOduration=1.560775571 podStartE2EDuration="4.093916519s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:49.74402004 +0000 UTC m=+126.619546189" lastFinishedPulling="2026-06-02 15:48:52.277160997 +0000 UTC m=+129.152687137" observedRunningTime="2026-06-02 15:48:53.090001788 +0000 UTC m=+129.965527948" watchObservedRunningTime="2026-06-02 15:48:53.093916519 +0000 UTC m=+129.969442682"
Jun 02 15:48:53.394157 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:53.394060    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:48:53.394323 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:53.394230    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:48:53.394406 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:53.394324    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs podName:e4ecbb10-dcd6-4827-93fc-7ffc14508a87 nodeName:}" failed. No retries permitted until 2026-06-02 15:50:55.394301445 +0000 UTC m=+252.269827584 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs") pod "network-metrics-daemon-9qqr9" (UID: "e4ecbb10-dcd6-4827-93fc-7ffc14508a87") : secret "metrics-daemon-secret" not found
Jun 02 15:48:54.120447 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.120412    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"]
Jun 02 15:48:54.123741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.123719    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"
Jun 02 15:48:54.127157 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.127136    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jun 02 15:48:54.127893 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.127876    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:54.127957 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.127881    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xdlq5\""
Jun 02 15:48:54.137384 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.137344    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"]
Jun 02 15:48:54.200538 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.200498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2qt\" (UniqueName: \"kubernetes.io/projected/55339f68-4ed2-4754-a3c5-a78690703a70-kube-api-access-8p2qt\") pod \"migrator-96c6f749-ktn6r\" (UID: \"55339f68-4ed2-4754-a3c5-a78690703a70\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"
Jun 02 15:48:54.300935 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.300885    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2qt\" (UniqueName: \"kubernetes.io/projected/55339f68-4ed2-4754-a3c5-a78690703a70-kube-api-access-8p2qt\") pod \"migrator-96c6f749-ktn6r\" (UID: \"55339f68-4ed2-4754-a3c5-a78690703a70\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"
Jun 02 15:48:54.309520 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.309488    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2qt\" (UniqueName: \"kubernetes.io/projected/55339f68-4ed2-4754-a3c5-a78690703a70-kube-api-access-8p2qt\") pod \"migrator-96c6f749-ktn6r\" (UID: \"55339f68-4ed2-4754-a3c5-a78690703a70\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"
Jun 02 15:48:54.433292 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.433187    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"
Jun 02 15:48:54.559650 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:54.559623    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r"]
Jun 02 15:48:54.562529 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:48:54.562500    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55339f68_4ed2_4754_a3c5_a78690703a70.slice/crio-d0f3da536c0f5c9098bb08277b4743b6360a34a5ee48325a7e5404ca6c4b5f11 WatchSource:0}: Error finding container d0f3da536c0f5c9098bb08277b4743b6360a34a5ee48325a7e5404ca6c4b5f11: Status 404 returned error can't find the container with id d0f3da536c0f5c9098bb08277b4743b6360a34a5ee48325a7e5404ca6c4b5f11
Jun 02 15:48:55.038996 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.038962    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r" event={"ID":"55339f68-4ed2-4754-a3c5-a78690703a70","Type":"ContainerStarted","Data":"d0f3da536c0f5c9098bb08277b4743b6360a34a5ee48325a7e5404ca6c4b5f11"}
Jun 02 15:48:55.107935 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.107897    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:55.108109 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.107955    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:48:55.108345 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.108047    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:49:11.108024407 +0000 UTC m=+147.983550555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:55.108495 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.108415    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:55.108656 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.108499    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs podName:34318a4f-dd28-449c-ae48-84eec2db293b nodeName:}" failed. No retries permitted until 2026-06-02 15:49:11.108473575 +0000 UTC m=+147.983999727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs") pod "router-default-6bc549f7b9-ncs76" (UID: "34318a4f-dd28-449c-ae48-84eec2db293b") : secret "router-metrics-certs-default" not found
Jun 02 15:48:55.136912 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.136877    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:48:55.140091 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.140065    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.142665 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.142646    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 02 15:48:55.142801 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.142693    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 02 15:48:55.143114 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.143089    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 02 15:48:55.143359 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.143343    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nsswx\""
Jun 02 15:48:55.148144 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.148122    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 02 15:48:55.152599 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.152579    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:48:55.209387 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209559 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209559 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209441    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209559 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209702 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209571    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:48:55.209702 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209702 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4rt\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209702 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209889 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.209720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.209889 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.209736    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:55.209889 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.209809    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls podName:9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a nodeName:}" failed. No retries permitted until 2026-06-02 15:49:11.209791068 +0000 UTC m=+148.085317212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-69b6cbdb88-qncnd" (UID: "9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a") : secret "cluster-monitoring-operator-tls" not found
Jun 02 15:48:55.310637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310547    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.310637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310598    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.310637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310618    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4rt\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.310637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310639    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310752    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310799    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310844    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.310870    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.310987    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:48:55.311035 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.311010    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f495554d6-ccwk5: secret "image-registry-tls" not found
Jun 02 15:48:55.311400 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.311080    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls podName:0f68cac5-7132-40fe-be82-52ca912bd98d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:55.811057683 +0000 UTC m=+132.686583837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls") pod "image-registry-f495554d6-ccwk5" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d") : secret "image-registry-tls" not found
Jun 02 15:48:55.311400 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.311118    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311504 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.311416    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.311712 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.311695    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.312906 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.312889    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.313055 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.313039    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.322182 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.322156    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.322837 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.322816    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4rt\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.814849 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:55.814812    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:55.815011 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.814973    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:48:55.815011 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.814994    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f495554d6-ccwk5: secret "image-registry-tls" not found
Jun 02 15:48:55.815076 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:55.815052    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls podName:0f68cac5-7132-40fe-be82-52ca912bd98d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:56.815036313 +0000 UTC m=+133.690562457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls") pod "image-registry-f495554d6-ccwk5" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d") : secret "image-registry-tls" not found
Jun 02 15:48:56.043663 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:56.043632    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r" event={"ID":"55339f68-4ed2-4754-a3c5-a78690703a70","Type":"ContainerStarted","Data":"27847358f03e0e2600535f976363d8b852d455a9fba52831e1d396762744ff50"}
Jun 02 15:48:56.043663 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:56.043666    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r" event={"ID":"55339f68-4ed2-4754-a3c5-a78690703a70","Type":"ContainerStarted","Data":"20c9bff6849273f2356583b01e2e8a756731061d7f703e47c5c40fd4105506fa"}
Jun 02 15:48:56.061451 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:56.061404    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-ktn6r" podStartSLOduration=0.782797986 podStartE2EDuration="2.061390068s" podCreationTimestamp="2026-06-02 15:48:54 +0000 UTC" firstStartedPulling="2026-06-02 15:48:54.564460281 +0000 UTC m=+131.439986420" lastFinishedPulling="2026-06-02 15:48:55.84305236 +0000 UTC m=+132.718578502" observedRunningTime="2026-06-02 15:48:56.06011703 +0000 UTC m=+132.935643192" watchObservedRunningTime="2026-06-02 15:48:56.061390068 +0000 UTC m=+132.936916228"
Jun 02 15:48:56.823722 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:56.823688    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:56.824142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:56.823842    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:48:56.824142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:56.823864    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f495554d6-ccwk5: secret "image-registry-tls" not found
Jun 02 15:48:56.824142 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:56.823918    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls podName:0f68cac5-7132-40fe-be82-52ca912bd98d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:58.823902412 +0000 UTC m=+135.699428551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls") pod "image-registry-f495554d6-ccwk5" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d") : secret "image-registry-tls" not found
Jun 02 15:48:57.126824 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:57.126735    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:48:57.126964 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:57.126883    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:57.126964 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:57.126939    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls podName:6fa6ba27-492c-4831-be65-53e14d4d825d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:05.126924757 +0000 UTC m=+142.002450896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-wzg84" (UID: "6fa6ba27-492c-4831-be65-53e14d4d825d") : secret "samples-operator-tls" not found
Jun 02 15:48:58.841668 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:48:58.841632    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:48:58.842153 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:58.841779    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:48:58.842153 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:58.841800    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f495554d6-ccwk5: secret "image-registry-tls" not found
Jun 02 15:48:58.842153 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:48:58.841866    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls podName:0f68cac5-7132-40fe-be82-52ca912bd98d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:02.841848213 +0000 UTC m=+139.717374372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls") pod "image-registry-f495554d6-ccwk5" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d") : secret "image-registry-tls" not found
Jun 02 15:49:02.877820 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:02.877770    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:02.878218 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:02.877923    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:49:02.878218 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:02.877941    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f495554d6-ccwk5: secret "image-registry-tls" not found
Jun 02 15:49:02.878218 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:02.877996    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls podName:0f68cac5-7132-40fe-be82-52ca912bd98d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:10.877979594 +0000 UTC m=+147.753506002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls") pod "image-registry-f495554d6-ccwk5" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d") : secret "image-registry-tls" not found
Jun 02 15:49:05.199535 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:05.199493    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:49:05.201776 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:05.201756    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ba27-492c-4831-be65-53e14d4d825d-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-wzg84\" (UID: \"6fa6ba27-492c-4831-be65-53e14d4d825d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:49:05.314637 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:05.314600    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"
Jun 02 15:49:05.435768 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:05.435728    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84"]
Jun 02 15:49:06.069725 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:06.069675    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84" event={"ID":"6fa6ba27-492c-4831-be65-53e14d4d825d","Type":"ContainerStarted","Data":"14c620aec68c44a72c893fc57258e59d2f32cd886b678e1d1eb5f46879dae23c"}
Jun 02 15:49:08.074716 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:08.074681    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84" event={"ID":"6fa6ba27-492c-4831-be65-53e14d4d825d","Type":"ContainerStarted","Data":"1a5bdfe84968feb468a36717ee1674805587b94cee63c608a6d9f67d84cce538"}
Jun 02 15:49:08.075160 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:08.074721    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84" event={"ID":"6fa6ba27-492c-4831-be65-53e14d4d825d","Type":"ContainerStarted","Data":"bdebffffb069b73a515f50af3681b8238e0ad355744fbc8f137dd301bf4b6ddb"}
Jun 02 15:49:08.119514 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:08.119461    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-wzg84" podStartSLOduration=17.507916523 podStartE2EDuration="19.119444043s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:49:05.47723878 +0000 UTC m=+142.352764918" lastFinishedPulling="2026-06-02 15:49:07.0887663 +0000 UTC m=+143.964292438" observedRunningTime="2026-06-02 15:49:08.117854193 +0000 UTC m=+144.993380353" watchObservedRunningTime="2026-06-02 15:49:08.119444043 +0000 UTC m=+144.994970204"
Jun 02 15:49:10.951634 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:10.951602    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:10.954065 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:10.954042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"image-registry-f495554d6-ccwk5\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") " pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:11.052545 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.052506    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:11.153078 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.153045    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:11.153243 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.153089    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:11.153672 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.153649    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34318a4f-dd28-449c-ae48-84eec2db293b-service-ca-bundle\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:11.155334 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.155312    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34318a4f-dd28-449c-ae48-84eec2db293b-metrics-certs\") pod \"router-default-6bc549f7b9-ncs76\" (UID: \"34318a4f-dd28-449c-ae48-84eec2db293b\") " pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:11.162918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.162898    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x8bn8\""
Jun 02 15:49:11.169244 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.169224    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:11.211838 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.211721    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:49:11.215252 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:11.215219    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f68cac5_7132_40fe_be82_52ca912bd98d.slice/crio-f66afc092e9cbb93d9bc60f9c7cc31987226fe773aa676080d1c1fa8182ce06a WatchSource:0}: Error finding container f66afc092e9cbb93d9bc60f9c7cc31987226fe773aa676080d1c1fa8182ce06a: Status 404 returned error can't find the container with id f66afc092e9cbb93d9bc60f9c7cc31987226fe773aa676080d1c1fa8182ce06a
Jun 02 15:49:11.254515 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.254480    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:49:11.257232 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.257204    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-69b6cbdb88-qncnd\" (UID: \"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a\") " pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:49:11.260801 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.260779    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-946qt\""
Jun 02 15:49:11.268182 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.267936    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"
Jun 02 15:49:11.312521 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.312495    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bc549f7b9-ncs76"]
Jun 02 15:49:11.315898 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:11.314734    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34318a4f_dd28_449c_ae48_84eec2db293b.slice/crio-3918cf537bc48deea26d9d17e4a9481b0ec3354342ac3a0557acdaf9b7d3b741 WatchSource:0}: Error finding container 3918cf537bc48deea26d9d17e4a9481b0ec3354342ac3a0557acdaf9b7d3b741: Status 404 returned error can't find the container with id 3918cf537bc48deea26d9d17e4a9481b0ec3354342ac3a0557acdaf9b7d3b741
Jun 02 15:49:11.408749 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:11.408715    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd"]
Jun 02 15:49:11.417059 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:11.417031    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9601e117_4d6e_4a2b_ba90_2b0fc86e4a6a.slice/crio-8216016ff9a9e173fe0eb58a19806c01b60cadc322adba7f1ddc612c6ce714e9 WatchSource:0}: Error finding container 8216016ff9a9e173fe0eb58a19806c01b60cadc322adba7f1ddc612c6ce714e9: Status 404 returned error can't find the container with id 8216016ff9a9e173fe0eb58a19806c01b60cadc322adba7f1ddc612c6ce714e9
Jun 02 15:49:12.084357 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.084316    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd" event={"ID":"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a","Type":"ContainerStarted","Data":"8216016ff9a9e173fe0eb58a19806c01b60cadc322adba7f1ddc612c6ce714e9"}
Jun 02 15:49:12.085406 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.085361    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bc549f7b9-ncs76" event={"ID":"34318a4f-dd28-449c-ae48-84eec2db293b","Type":"ContainerStarted","Data":"f8737166adb74e336e7003445a4fe962a01f54ccc0b92261a3070dd2f37478d1"}
Jun 02 15:49:12.085406 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.085405    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bc549f7b9-ncs76" event={"ID":"34318a4f-dd28-449c-ae48-84eec2db293b","Type":"ContainerStarted","Data":"3918cf537bc48deea26d9d17e4a9481b0ec3354342ac3a0557acdaf9b7d3b741"}
Jun 02 15:49:12.086507 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.086485    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" event={"ID":"0f68cac5-7132-40fe-be82-52ca912bd98d","Type":"ContainerStarted","Data":"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"}
Jun 02 15:49:12.086598 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.086509    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" event={"ID":"0f68cac5-7132-40fe-be82-52ca912bd98d","Type":"ContainerStarted","Data":"f66afc092e9cbb93d9bc60f9c7cc31987226fe773aa676080d1c1fa8182ce06a"}
Jun 02 15:49:12.086598 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.086593    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:12.114115 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.114068    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-6bc549f7b9-ncs76" podStartSLOduration=33.114049294 podStartE2EDuration="33.114049294s" podCreationTimestamp="2026-06-02 15:48:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:12.11022686 +0000 UTC m=+148.985753021" watchObservedRunningTime="2026-06-02 15:49:12.114049294 +0000 UTC m=+148.989575458"
Jun 02 15:49:12.170412 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.170360    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:12.173155 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.173120    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:12.203054 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:12.202998    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" podStartSLOduration=17.202979767 podStartE2EDuration="17.202979767s" podCreationTimestamp="2026-06-02 15:48:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:12.145927364 +0000 UTC m=+149.021453525" watchObservedRunningTime="2026-06-02 15:49:12.202979767 +0000 UTC m=+149.078505928"
Jun 02 15:49:13.089198 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:13.089171    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:13.090346 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:13.090324    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bc549f7b9-ncs76"
Jun 02 15:49:14.092728 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:14.092688    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd" event={"ID":"9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a","Type":"ContainerStarted","Data":"c32ccddb252f9970d9963590303c30ff4f3eb0ef55a126bb2fb708ac6962f2bb"}
Jun 02 15:49:14.116215 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:14.116164    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-69b6cbdb88-qncnd" podStartSLOduration=33.47702315 podStartE2EDuration="35.116145387s" podCreationTimestamp="2026-06-02 15:48:39 +0000 UTC" firstStartedPulling="2026-06-02 15:49:11.418934456 +0000 UTC m=+148.294460595" lastFinishedPulling="2026-06-02 15:49:13.058056693 +0000 UTC m=+149.933582832" observedRunningTime="2026-06-02 15:49:14.11555917 +0000 UTC m=+150.991085328" watchObservedRunningTime="2026-06-02 15:49:14.116145387 +0000 UTC m=+150.991671551"
Jun 02 15:49:17.435666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.435628    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:49:17.611336 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.611217    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jdd6d"]
Jun 02 15:49:17.614570 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.614546    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.624073 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:17.624046    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-131-30.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-30.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jun 02 15:49:17.630509 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.630491    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:49:17.630672 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.630655    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cr5dk\""
Jun 02 15:49:17.641427 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.641397    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdd6d"]
Jun 02 15:49:17.711784 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.711743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-crio-socket\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.712085 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.712067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-data-volume\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.712203 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.712188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.712334 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.712320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.712480 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.712466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbq4p\" (UniqueName: \"kubernetes.io/projected/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-api-access-bbq4p\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813024 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.812989    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813178 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813041    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bbq4p\" (UniqueName: \"kubernetes.io/projected/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-api-access-bbq4p\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813166    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-crio-socket\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813211    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-data-volume\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813265 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813243    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813430 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813258    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-crio-socket\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813629 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813610    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-data-volume\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.813817 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.813800    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:17.865267 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:17.865188    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbq4p\" (UniqueName: \"kubernetes.io/projected/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-kube-api-access-bbq4p\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:18.459476 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:18.459446    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:49:18.465642 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:18.465622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ae18564-cb3d-4791-95d7-5e0f3b1e3822-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdd6d\" (UID: \"3ae18564-cb3d-4791-95d7-5e0f3b1e3822\") " pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:18.523680 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:18.523641    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdd6d"
Jun 02 15:49:18.675889 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:18.675857    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdd6d"]
Jun 02 15:49:18.679060 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:18.679032    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae18564_cb3d_4791_95d7_5e0f3b1e3822.slice/crio-fde070edc52378ec55decc58b00dc99325175e15ab764953cfd72eaf30683d27 WatchSource:0}: Error finding container fde070edc52378ec55decc58b00dc99325175e15ab764953cfd72eaf30683d27: Status 404 returned error can't find the container with id fde070edc52378ec55decc58b00dc99325175e15ab764953cfd72eaf30683d27
Jun 02 15:49:19.106327 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:19.106238    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdd6d" event={"ID":"3ae18564-cb3d-4791-95d7-5e0f3b1e3822","Type":"ContainerStarted","Data":"62799691ed963a820075c940e751a6a639f8bcfafc781a9c7492dc4db818628e"}
Jun 02 15:49:19.106327 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:19.106276    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdd6d" event={"ID":"3ae18564-cb3d-4791-95d7-5e0f3b1e3822","Type":"ContainerStarted","Data":"fde070edc52378ec55decc58b00dc99325175e15ab764953cfd72eaf30683d27"}
Jun 02 15:49:20.110770 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:20.110739    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdd6d" event={"ID":"3ae18564-cb3d-4791-95d7-5e0f3b1e3822","Type":"ContainerStarted","Data":"ff13176009c3ba865b78563b7abdbaa2a03f53ca44db76edcb31dba419a3e79d"}
Jun 02 15:49:21.114829 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:21.114781    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdd6d" event={"ID":"3ae18564-cb3d-4791-95d7-5e0f3b1e3822","Type":"ContainerStarted","Data":"6fc59bec7efe4a7c5f7ef2f8606cfbbaf9d3af84972417ea9496437baf457fed"}
Jun 02 15:49:21.139983 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:21.139933    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jdd6d" podStartSLOduration=2.2088119 podStartE2EDuration="4.13991933s" podCreationTimestamp="2026-06-02 15:49:17 +0000 UTC" firstStartedPulling="2026-06-02 15:49:18.72944749 +0000 UTC m=+155.604973628" lastFinishedPulling="2026-06-02 15:49:20.660554916 +0000 UTC m=+157.536081058" observedRunningTime="2026-06-02 15:49:21.139221795 +0000 UTC m=+158.014747957" watchObservedRunningTime="2026-06-02 15:49:21.13991933 +0000 UTC m=+158.015445484"
Jun 02 15:49:21.617274 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:21.617230    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v22gh" podUID="b0d74645-7b90-430f-af77-f554a9418180"
Jun 02 15:49:21.631535 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:21.631504    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-5wfsn" podUID="e3b6c156-e1a0-4290-9a47-55dd7016276e"
Jun 02 15:49:21.691105 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:21.691063    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9qqr9" podUID="e4ecbb10-dcd6-4827-93fc-7ffc14508a87"
Jun 02 15:49:22.117691 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:22.117661    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:22.117691 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:22.117662    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:49:25.209229 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.209194    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"]
Jun 02 15:49:25.212131 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.212111    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.215222 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.215169    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jun 02 15:49:25.215340 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.215247    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:49:25.216138 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.216122    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7k7jc\""
Jun 02 15:49:25.216185 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.216146    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:49:25.227410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.227385    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"]
Jun 02 15:49:25.241078 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.241050    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"]
Jun 02 15:49:25.243893 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.243874    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.251465 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.251425    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 02 15:49:25.251590 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.251509    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:49:25.251772 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.251756    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gczjd\""
Jun 02 15:49:25.251905 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.251889    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 02 15:49:25.257861 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.257843    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tzv6p"]
Jun 02 15:49:25.259967 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.259951    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.265324 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.265304    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:49:25.265584 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.265568    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:49:25.265681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.265599    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:49:25.265681 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.265618    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mnlvm\""
Jun 02 15:49:25.267141 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.267117    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"]
Jun 02 15:49:25.274348 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274330    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf11c0b-146e-4805-8b4e-a231d2033047-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.274438 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274438 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xmv\" (UniqueName: \"kubernetes.io/projected/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-api-access-m2xmv\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274516 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.274516 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a08ddb5-3017-476a-9257-e4acc59fa0c3-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274516 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274496    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274613 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.274613 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274613 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.274706 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.274656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29bv\" (UniqueName: \"kubernetes.io/projected/cdf11c0b-146e-4805-8b4e-a231d2033047-kube-api-access-t29bv\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.375221 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375187    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.375221 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375226    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a08ddb5-3017-476a-9257-e4acc59fa0c3-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375264    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnds\" (UniqueName: \"kubernetes.io/projected/e317b143-129b-4b87-b378-459f58efb816-kube-api-access-xjnds\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.375346    2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.375362    2571 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375389    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375417    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.375438    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls podName:cdf11c0b-146e-4805-8b4e-a231d2033047 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:25.875417384 +0000 UTC m=+162.750943524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls") pod "openshift-state-metrics-85658796db-wqgmk" (UID: "cdf11c0b-146e-4805-8b4e-a231d2033047") : secret "openshift-state-metrics-tls" not found
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.375470    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls podName:5a08ddb5-3017-476a-9257-e4acc59fa0c3 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:25.875459526 +0000 UTC m=+162.750985664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls") pod "kube-state-metrics-8469c486bb-4lqzx" (UID: "5a08ddb5-3017-476a-9257-e4acc59fa0c3") : secret "kube-state-metrics-tls" not found
Jun 02 15:49:25.375513 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-node-exporter-wtmp\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375552    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-root\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375645    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t29bv\" (UniqueName: \"kubernetes.io/projected/cdf11c0b-146e-4805-8b4e-a231d2033047-kube-api-access-t29bv\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375672    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e317b143-129b-4b87-b378-459f58efb816-node-exporter-textfile\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375694    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a08ddb5-3017-476a-9257-e4acc59fa0c3-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-sys\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375806    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf11c0b-146e-4805-8b4e-a231d2033047-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375893    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.375918 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375920    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xmv\" (UniqueName: \"kubernetes.io/projected/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-api-access-m2xmv\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.376395 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.375956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-metrics-client-ca\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.376395 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.376094    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.376395 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.376222    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.376497 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.376465    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf11c0b-146e-4805-8b4e-a231d2033047-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.377892 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.377870    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.378087 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.378072    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.398968 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.398946    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xmv\" (UniqueName: \"kubernetes.io/projected/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-api-access-m2xmv\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.401443 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.401421    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29bv\" (UniqueName: \"kubernetes.io/projected/cdf11c0b-146e-4805-8b4e-a231d2033047-kube-api-access-t29bv\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.476497 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476461    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-metrics-client-ca\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476557    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnds\" (UniqueName: \"kubernetes.io/projected/e317b143-129b-4b87-b378-459f58efb816-kube-api-access-xjnds\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476594    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-node-exporter-wtmp\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476620    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-root\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476645    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476677    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e317b143-129b-4b87-b378-459f58efb816-node-exporter-textfile\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476700    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-sys\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476701    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-root\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476726    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-node-exporter-wtmp\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476780    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.476798    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e317b143-129b-4b87-b378-459f58efb816-sys\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.476948 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.476887    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 02 15:49:25.477227 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:25.476960    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls podName:e317b143-129b-4b87-b378-459f58efb816 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:25.976941855 +0000 UTC m=+162.852467995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls") pod "node-exporter-tzv6p" (UID: "e317b143-129b-4b87-b378-459f58efb816") : secret "node-exporter-tls" not found
Jun 02 15:49:25.477227 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.477014    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e317b143-129b-4b87-b378-459f58efb816-node-exporter-textfile\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.477227 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.477036    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-metrics-client-ca\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.477227 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.477217    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e317b143-129b-4b87-b378-459f58efb816-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.479041 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.479019    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.495053 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.495027    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnds\" (UniqueName: \"kubernetes.io/projected/e317b143-129b-4b87-b378-459f58efb816-kube-api-access-xjnds\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.882030 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.881941    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.882030 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.881981    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.884344 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.884319    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdf11c0b-146e-4805-8b4e-a231d2033047-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-wqgmk\" (UID: \"cdf11c0b-146e-4805-8b4e-a231d2033047\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:25.884474 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.884327    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a08ddb5-3017-476a-9257-e4acc59fa0c3-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-4lqzx\" (UID: \"5a08ddb5-3017-476a-9257-e4acc59fa0c3\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:25.982459 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.982419    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:25.984813 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:25.984791    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e317b143-129b-4b87-b378-459f58efb816-node-exporter-tls\") pod \"node-exporter-tzv6p\" (UID: \"e317b143-129b-4b87-b378-459f58efb816\") " pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:26.120994 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.120946    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"
Jun 02 15:49:26.153290 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.153210    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"
Jun 02 15:49:26.169262 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.169224    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzv6p"
Jun 02 15:49:26.180320 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:26.180286    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode317b143_129b_4b87_b378_459f58efb816.slice/crio-b39e35a01d3d76e552325d00a86d7c6c3b08f6fc699e7868a2c9a71fd33123eb WatchSource:0}: Error finding container b39e35a01d3d76e552325d00a86d7c6c3b08f6fc699e7868a2c9a71fd33123eb: Status 404 returned error can't find the container with id b39e35a01d3d76e552325d00a86d7c6c3b08f6fc699e7868a2c9a71fd33123eb
Jun 02 15:49:26.270536 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.270501    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-wqgmk"]
Jun 02 15:49:26.272577 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:26.272551    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf11c0b_146e_4805_8b4e_a231d2033047.slice/crio-d978d5adb5cbaf396cb70916de8c8713adce64e9b11ea3726432eb16facebb13 WatchSource:0}: Error finding container d978d5adb5cbaf396cb70916de8c8713adce64e9b11ea3726432eb16facebb13: Status 404 returned error can't find the container with id d978d5adb5cbaf396cb70916de8c8713adce64e9b11ea3726432eb16facebb13
Jun 02 15:49:26.306966 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.306942    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx"]
Jun 02 15:49:26.309975 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:26.309947    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a08ddb5_3017_476a_9257_e4acc59fa0c3.slice/crio-119aaeaeb4902763ecc0954d4ce2af95944b08afed568f8019976f87710a4f19 WatchSource:0}: Error finding container 119aaeaeb4902763ecc0954d4ce2af95944b08afed568f8019976f87710a4f19: Status 404 returned error can't find the container with id 119aaeaeb4902763ecc0954d4ce2af95944b08afed568f8019976f87710a4f19
Jun 02 15:49:26.487325 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.487291    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:49:26.487325 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.487341    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:26.489655 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.489627    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0d74645-7b90-430f-af77-f554a9418180-metrics-tls\") pod \"dns-default-v22gh\" (UID: \"b0d74645-7b90-430f-af77-f554a9418180\") " pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:26.489834 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.489821    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3b6c156-e1a0-4290-9a47-55dd7016276e-cert\") pod \"ingress-canary-5wfsn\" (UID: \"e3b6c156-e1a0-4290-9a47-55dd7016276e\") " pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:49:26.622921 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.622892    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ht87l\""
Jun 02 15:49:26.623131 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.623039    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cw7rp\""
Jun 02 15:49:26.629933 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.629914    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:26.630031 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.629942    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5wfsn"
Jun 02 15:49:26.794155 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.794124    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v22gh"]
Jun 02 15:49:26.798764 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:26.798734    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d74645_7b90_430f_af77_f554a9418180.slice/crio-4524781bb7cb3084981b89a2860032ea6292bcab04ec525a357c4c0d1f3ade0a WatchSource:0}: Error finding container 4524781bb7cb3084981b89a2860032ea6292bcab04ec525a357c4c0d1f3ade0a: Status 404 returned error can't find the container with id 4524781bb7cb3084981b89a2860032ea6292bcab04ec525a357c4c0d1f3ade0a
Jun 02 15:49:26.803805 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:26.803781    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5wfsn"]
Jun 02 15:49:26.807431 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:49:26.807404    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b6c156_e1a0_4290_9a47_55dd7016276e.slice/crio-7d8fb78e2fed6a416f3cfb8ee8814e3dfaecc2d57be8026fb8d956a306a89c74 WatchSource:0}: Error finding container 7d8fb78e2fed6a416f3cfb8ee8814e3dfaecc2d57be8026fb8d956a306a89c74: Status 404 returned error can't find the container with id 7d8fb78e2fed6a416f3cfb8ee8814e3dfaecc2d57be8026fb8d956a306a89c74
Jun 02 15:49:27.132204 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.132153    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5wfsn" event={"ID":"e3b6c156-e1a0-4290-9a47-55dd7016276e","Type":"ContainerStarted","Data":"7d8fb78e2fed6a416f3cfb8ee8814e3dfaecc2d57be8026fb8d956a306a89c74"}
Jun 02 15:49:27.133643 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.133611    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v22gh" event={"ID":"b0d74645-7b90-430f-af77-f554a9418180","Type":"ContainerStarted","Data":"4524781bb7cb3084981b89a2860032ea6292bcab04ec525a357c4c0d1f3ade0a"}
Jun 02 15:49:27.135092 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.135063    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzv6p" event={"ID":"e317b143-129b-4b87-b378-459f58efb816","Type":"ContainerStarted","Data":"b39e35a01d3d76e552325d00a86d7c6c3b08f6fc699e7868a2c9a71fd33123eb"}
Jun 02 15:49:27.137286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.137217    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk" event={"ID":"cdf11c0b-146e-4805-8b4e-a231d2033047","Type":"ContainerStarted","Data":"bdbc95f1d6a3c7cd4010ed847a84a48679a60788b123d07ec90989e32d2cd4cd"}
Jun 02 15:49:27.137286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.137248    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk" event={"ID":"cdf11c0b-146e-4805-8b4e-a231d2033047","Type":"ContainerStarted","Data":"1f8f813f5371b68e863a1c57fdca7a741e4b81b17d8fb09474be217a8128eb91"}
Jun 02 15:49:27.137286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.137262    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk" event={"ID":"cdf11c0b-146e-4805-8b4e-a231d2033047","Type":"ContainerStarted","Data":"d978d5adb5cbaf396cb70916de8c8713adce64e9b11ea3726432eb16facebb13"}
Jun 02 15:49:27.139419 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.139329    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx" event={"ID":"5a08ddb5-3017-476a-9257-e4acc59fa0c3","Type":"ContainerStarted","Data":"119aaeaeb4902763ecc0954d4ce2af95944b08afed568f8019976f87710a4f19"}
Jun 02 15:49:27.444782 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.444414    2571 patch_prober.go:28] interesting pod/image-registry-f495554d6-ccwk5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 02 15:49:27.444782 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:27.444491    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:49:28.145200 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:28.145164    2571 generic.go:358] "Generic (PLEG): container finished" podID="e317b143-129b-4b87-b378-459f58efb816" containerID="d4aa6f87c2a549aa6afc650b258193e55978d0e9ca2cdc2873bde3b0a09e0b19" exitCode=0
Jun 02 15:49:28.145390 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:28.145284    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzv6p" event={"ID":"e317b143-129b-4b87-b378-459f58efb816","Type":"ContainerDied","Data":"d4aa6f87c2a549aa6afc650b258193e55978d0e9ca2cdc2873bde3b0a09e0b19"}
Jun 02 15:49:29.154917 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.154873    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5wfsn" event={"ID":"e3b6c156-e1a0-4290-9a47-55dd7016276e","Type":"ContainerStarted","Data":"55eb96c9bd706419da9cdbf564576962362ca4235dd18251da68b1162dbb9dc8"}
Jun 02 15:49:29.157595 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.157524    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v22gh" event={"ID":"b0d74645-7b90-430f-af77-f554a9418180","Type":"ContainerStarted","Data":"bb145905d35693455da04958f37b0ef17601b4b4ce9ddd7020615b24f3d7ca4d"}
Jun 02 15:49:29.163579 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.162651    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzv6p" event={"ID":"e317b143-129b-4b87-b378-459f58efb816","Type":"ContainerStarted","Data":"201e97da004f9205036b16b513cd801fdd091a390733b6e995856f1b5d4e4b73"}
Jun 02 15:49:29.166581 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.164977    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk" event={"ID":"cdf11c0b-146e-4805-8b4e-a231d2033047","Type":"ContainerStarted","Data":"ea44855b78710b0c8091be6dd251accb232cd407e2bbac6b2948a2e581ab3ff9"}
Jun 02 15:49:29.166822 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.166761    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx" event={"ID":"5a08ddb5-3017-476a-9257-e4acc59fa0c3","Type":"ContainerStarted","Data":"9388483fe1e8adf5d1b16b0a113d5a56713c051a5fa5da55c1e25acfd8b95272"}
Jun 02 15:49:29.237184 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.236942    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-85658796db-wqgmk" podStartSLOduration=1.643047845 podStartE2EDuration="4.236920609s" podCreationTimestamp="2026-06-02 15:49:25 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.423277419 +0000 UTC m=+163.298803558" lastFinishedPulling="2026-06-02 15:49:29.017150169 +0000 UTC m=+165.892676322" observedRunningTime="2026-06-02 15:49:29.236284299 +0000 UTC m=+166.111810460" watchObservedRunningTime="2026-06-02 15:49:29.236920609 +0000 UTC m=+166.112446770"
Jun 02 15:49:29.237410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:29.237354    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5wfsn" podStartSLOduration=129.019945417 podStartE2EDuration="2m11.237342436s" podCreationTimestamp="2026-06-02 15:47:18 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.809695488 +0000 UTC m=+163.685221633" lastFinishedPulling="2026-06-02 15:49:29.027092501 +0000 UTC m=+165.902618652" observedRunningTime="2026-06-02 15:49:29.194259344 +0000 UTC m=+166.069785509" watchObservedRunningTime="2026-06-02 15:49:29.237342436 +0000 UTC m=+166.112868592"
Jun 02 15:49:30.170871 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.170821    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v22gh" event={"ID":"b0d74645-7b90-430f-af77-f554a9418180","Type":"ContainerStarted","Data":"922bf2d808b114d6efe6a7ae876e9313b7cc142ec2d332d0b95635f46f637c23"}
Jun 02 15:49:30.171284 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.170965    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:30.172587 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.172565    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzv6p" event={"ID":"e317b143-129b-4b87-b378-459f58efb816","Type":"ContainerStarted","Data":"1af7dbcbef110ed77026b3af337a66e7f22967ce0c3c953f3b5acbb88a2aefe7"}
Jun 02 15:49:30.174308 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.174287    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx" event={"ID":"5a08ddb5-3017-476a-9257-e4acc59fa0c3","Type":"ContainerStarted","Data":"719af44203ae0eecd938cba4b67d4fb9f98930ec9c65942fb50c0caea7bda0d5"}
Jun 02 15:49:30.174403 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.174313    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx" event={"ID":"5a08ddb5-3017-476a-9257-e4acc59fa0c3","Type":"ContainerStarted","Data":"94572992f6578943055ce13dc82224cddee68a3a52e6ede133628b95321f7c0f"}
Jun 02 15:49:30.200194 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.200143    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-v22gh" podStartSLOduration=129.978782336 podStartE2EDuration="2m12.200129606s" podCreationTimestamp="2026-06-02 15:47:18 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.80119875 +0000 UTC m=+163.676724905" lastFinishedPulling="2026-06-02 15:49:29.022546022 +0000 UTC m=+165.898072175" observedRunningTime="2026-06-02 15:49:30.199810281 +0000 UTC m=+167.075336448" watchObservedRunningTime="2026-06-02 15:49:30.200129606 +0000 UTC m=+167.075655822"
Jun 02 15:49:30.269059 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.269004    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-8469c486bb-4lqzx" podStartSLOduration=2.559343381 podStartE2EDuration="5.268988694s" podCreationTimestamp="2026-06-02 15:49:25 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.311767293 +0000 UTC m=+163.187293445" lastFinishedPulling="2026-06-02 15:49:29.021412615 +0000 UTC m=+165.896938758" observedRunningTime="2026-06-02 15:49:30.268763485 +0000 UTC m=+167.144289648" watchObservedRunningTime="2026-06-02 15:49:30.268988694 +0000 UTC m=+167.144514855"
Jun 02 15:49:30.269241 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:30.269102    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tzv6p" podStartSLOduration=4.261696297 podStartE2EDuration="5.269097127s" podCreationTimestamp="2026-06-02 15:49:25 +0000 UTC" firstStartedPulling="2026-06-02 15:49:26.182658217 +0000 UTC m=+163.058184370" lastFinishedPulling="2026-06-02 15:49:27.190059059 +0000 UTC m=+164.065585200" observedRunningTime="2026-06-02 15:49:30.240388152 +0000 UTC m=+167.115914321" watchObservedRunningTime="2026-06-02 15:49:30.269097127 +0000 UTC m=+167.144623288"
Jun 02 15:49:33.683445 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:33.683410    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:49:37.440030 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:37.439999    2571 patch_prober.go:28] interesting pod/image-registry-f495554d6-ccwk5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 02 15:49:37.440424 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:37.440050    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:49:40.180356 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:40.180316    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v22gh"
Jun 02 15:49:42.455065 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.455026    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry" containerID="cri-o://a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a" gracePeriod=30
Jun 02 15:49:42.696799 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.696774    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:42.835604 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835570    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx4rt\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835628    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835658    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835682    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835704    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835731    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835749    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.835794 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.835790    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets\") pod \"0f68cac5-7132-40fe-be82-52ca912bd98d\" (UID: \"0f68cac5-7132-40fe-be82-52ca912bd98d\") "
Jun 02 15:49:42.836566 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.836517    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:42.836784 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.836741    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:42.838195 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.838149    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:42.838195 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.838159    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:42.838319 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.838195    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt" (OuterVolumeSpecName: "kube-api-access-vx4rt") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "kube-api-access-vx4rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:42.838319 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.838296    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:42.838319 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.838304    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:42.844854 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.844832    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0f68cac5-7132-40fe-be82-52ca912bd98d" (UID: "0f68cac5-7132-40fe-be82-52ca912bd98d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:49:42.937307 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937273    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-image-registry-private-configuration\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937307 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937304    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937307 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937314    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-registry-certificates\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937532 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937324    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-bound-sa-token\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937532 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937333    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f68cac5-7132-40fe-be82-52ca912bd98d-trusted-ca\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937532 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937341    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f68cac5-7132-40fe-be82-52ca912bd98d-installation-pull-secrets\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937532 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937350    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4rt\" (UniqueName: \"kubernetes.io/projected/0f68cac5-7132-40fe-be82-52ca912bd98d-kube-api-access-vx4rt\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:42.937532 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:42.937360    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f68cac5-7132-40fe-be82-52ca912bd98d-ca-trust-extracted\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:49:43.213876 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.213845    2571 generic.go:358] "Generic (PLEG): container finished" podID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerID="a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a" exitCode=0
Jun 02 15:49:43.214070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.213883    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f495554d6-ccwk5"
Jun 02 15:49:43.214070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.213905    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" event={"ID":"0f68cac5-7132-40fe-be82-52ca912bd98d","Type":"ContainerDied","Data":"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"}
Jun 02 15:49:43.214070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.213949    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f495554d6-ccwk5" event={"ID":"0f68cac5-7132-40fe-be82-52ca912bd98d","Type":"ContainerDied","Data":"f66afc092e9cbb93d9bc60f9c7cc31987226fe773aa676080d1c1fa8182ce06a"}
Jun 02 15:49:43.214070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.213968    2571 scope.go:117] "RemoveContainer" containerID="a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"
Jun 02 15:49:43.223339 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.223323    2571 scope.go:117] "RemoveContainer" containerID="a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"
Jun 02 15:49:43.223603 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:49:43.223578    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a\": container with ID starting with a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a not found: ID does not exist" containerID="a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"
Jun 02 15:49:43.223658 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.223612    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a"} err="failed to get container status \"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a\": rpc error: code = NotFound desc = could not find container \"a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a\": container with ID starting with a3a62e84ccd106f9519d7e360dd4b80d56eae536aa8537a821b1e70248a22e8a not found: ID does not exist"
Jun 02 15:49:43.245530 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.243350    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:49:43.251339 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.251317    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f495554d6-ccwk5"]
Jun 02 15:49:43.683953 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:49:43.683918    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" path="/var/lib/kubelet/pods/0f68cac5-7132-40fe-be82-52ca912bd98d/volumes"
Jun 02 15:50:09.285080 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:09.285045    2571 generic.go:358] "Generic (PLEG): container finished" podID="8f118d48-4f19-4dc1-a22a-b456f271c48c" containerID="6d96d8a79ef28529027eaf9c2acb67a1cf28caaa2b84b5eb867be6ec33ec0ba6" exitCode=0
Jun 02 15:50:09.285502 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:09.285117    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq" event={"ID":"8f118d48-4f19-4dc1-a22a-b456f271c48c","Type":"ContainerDied","Data":"6d96d8a79ef28529027eaf9c2acb67a1cf28caaa2b84b5eb867be6ec33ec0ba6"}
Jun 02 15:50:09.285502 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:09.285448    2571 scope.go:117] "RemoveContainer" containerID="6d96d8a79ef28529027eaf9c2acb67a1cf28caaa2b84b5eb867be6ec33ec0ba6"
Jun 02 15:50:10.289510 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:10.289477    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn2bq" event={"ID":"8f118d48-4f19-4dc1-a22a-b456f271c48c","Type":"ContainerStarted","Data":"d5ca5df9798eff71d94da15115009dcfe25227b4a347fa0dde2c51632125d872"}
Jun 02 15:50:17.307734 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:17.307701    2571 generic.go:358] "Generic (PLEG): container finished" podID="2db81b66-10f3-4fc9-a130-484b26440831" containerID="028a8ec42a9d3fde6ef179973e6ed356e3f6eeea83306158e662e9d9e9fed7d5" exitCode=0
Jun 02 15:50:17.308120 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:17.307776    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-c2bl2" event={"ID":"2db81b66-10f3-4fc9-a130-484b26440831","Type":"ContainerDied","Data":"028a8ec42a9d3fde6ef179973e6ed356e3f6eeea83306158e662e9d9e9fed7d5"}
Jun 02 15:50:17.308120 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:17.308103    2571 scope.go:117] "RemoveContainer" containerID="028a8ec42a9d3fde6ef179973e6ed356e3f6eeea83306158e662e9d9e9fed7d5"
Jun 02 15:50:18.313090 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:18.313048    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-c2bl2" event={"ID":"2db81b66-10f3-4fc9-a130-484b26440831","Type":"ContainerStarted","Data":"cbdccf8bc5b21fc328eedd34f2fc5a4e451cfd4561e1c5d454f2aedd5b32a666"}
Jun 02 15:50:19.363546 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:19.363515    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-69b6cbdb88-qncnd_9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a/cluster-monitoring-operator/0.log"
Jun 02 15:50:19.523808 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:19.523781    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-state-metrics/0.log"
Jun 02 15:50:19.722286 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:19.722261    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-rbac-proxy-main/0.log"
Jun 02 15:50:19.926696 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:19.926666    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-rbac-proxy-self/0.log"
Jun 02 15:50:21.725653 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:21.725626    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/init-textfile/0.log"
Jun 02 15:50:21.925641 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:21.925618    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/node-exporter/0.log"
Jun 02 15:50:22.123310 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:22.123231    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/kube-rbac-proxy/0.log"
Jun 02 15:50:22.325829 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:22.325799    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/kube-rbac-proxy-main/0.log"
Jun 02 15:50:22.524219 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:22.524188    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/kube-rbac-proxy-self/0.log"
Jun 02 15:50:22.732512 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:22.732459    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/openshift-state-metrics/0.log"
Jun 02 15:50:23.328772 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:23.328742    2571 generic.go:358] "Generic (PLEG): container finished" podID="36b97419-e9e4-4f02-b246-80248eeb0025" containerID="6abf3f7a6063bbbc43fe1e7b05f5ea1383a27e0e928df4c59f892905d58a8e92" exitCode=0
Jun 02 15:50:23.328937 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:23.328813    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw" event={"ID":"36b97419-e9e4-4f02-b246-80248eeb0025","Type":"ContainerDied","Data":"6abf3f7a6063bbbc43fe1e7b05f5ea1383a27e0e928df4c59f892905d58a8e92"}
Jun 02 15:50:23.329129 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:23.329115    2571 scope.go:117] "RemoveContainer" containerID="6abf3f7a6063bbbc43fe1e7b05f5ea1383a27e0e928df4c59f892905d58a8e92"
Jun 02 15:50:24.332780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:24.332744    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-7wsjw" event={"ID":"36b97419-e9e4-4f02-b246-80248eeb0025","Type":"ContainerStarted","Data":"c6e40686e38f88ec079c90095a80c97f82ca0900da35c08d49b5ef9ca627147c"}
Jun 02 15:50:24.929038 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:24.929008    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bc549f7b9-ncs76_34318a4f-dd28-449c-ae48-84eec2db293b/router/0.log"
Jun 02 15:50:25.127185 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:25.127155    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5wfsn_e3b6c156-e1a0-4290-9a47-55dd7016276e/serve-healthcheck-canary/0.log"
Jun 02 15:50:37.335564 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.335529    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:50:37.335960 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.335835    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry"
Jun 02 15:50:37.335960 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.335848    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry"
Jun 02 15:50:37.335960 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.335894    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f68cac5-7132-40fe-be82-52ca912bd98d" containerName="registry"
Jun 02 15:50:37.338821 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.338799    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.342202 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.342180    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 02 15:50:37.342973 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.342950    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 02 15:50:37.343160 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343145    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 02 15:50:37.343160 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343156    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 02 15:50:37.343282 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343263    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7qwlp\""
Jun 02 15:50:37.343319 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343269    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 02 15:50:37.343410 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343395    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 02 15:50:37.343491 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.343476    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 02 15:50:37.350458 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.350439    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:50:37.489037 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.488994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.489206 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.489046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.489206 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.489107    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.489206 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.489142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.489206 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.489186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.489337 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.489256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt56g\" (UniqueName: \"kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.589952 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.589866    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.589952 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.589921    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.589952 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.589959    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590122 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.589981    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590122 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.590033    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590122 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.590060    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zt56g\" (UniqueName: \"kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590795 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.590772    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590919 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.590836    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.590919 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.590838    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.592428 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.592402    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.592514 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.592405    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.602407 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.602386    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt56g\" (UniqueName: \"kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g\") pod \"console-785644856d-rbvqj\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") " pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.648423 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.648347    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:37.811274 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:37.811242    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:50:37.814546 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:50:37.814521    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c23798_bc35_4951_855c_3905f42a8d0f.slice/crio-875de03fcf50166d8cd84fe9fbdb49139026a07030a141fad499851b383d6bb6 WatchSource:0}: Error finding container 875de03fcf50166d8cd84fe9fbdb49139026a07030a141fad499851b383d6bb6: Status 404 returned error can't find the container with id 875de03fcf50166d8cd84fe9fbdb49139026a07030a141fad499851b383d6bb6
Jun 02 15:50:38.374875 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:38.374835    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785644856d-rbvqj" event={"ID":"17c23798-bc35-4951-855c-3905f42a8d0f","Type":"ContainerStarted","Data":"875de03fcf50166d8cd84fe9fbdb49139026a07030a141fad499851b383d6bb6"}
Jun 02 15:50:41.391151 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:41.391113    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785644856d-rbvqj" event={"ID":"17c23798-bc35-4951-855c-3905f42a8d0f","Type":"ContainerStarted","Data":"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"}
Jun 02 15:50:47.649446 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:47.649399    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:47.649900 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:47.649465    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:47.654266 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:47.654235    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:47.700885 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:47.700838    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-785644856d-rbvqj" podStartSLOduration=7.986119689 podStartE2EDuration="10.700825329s" podCreationTimestamp="2026-06-02 15:50:37 +0000 UTC" firstStartedPulling="2026-06-02 15:50:37.816414658 +0000 UTC m=+234.691940801" lastFinishedPulling="2026-06-02 15:50:40.531120302 +0000 UTC m=+237.406646441" observedRunningTime="2026-06-02 15:50:41.420867698 +0000 UTC m=+238.296393870" watchObservedRunningTime="2026-06-02 15:50:47.700825329 +0000 UTC m=+244.576351489"
Jun 02 15:50:48.413598 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:48.413570    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:50:49.620352 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:49.620321    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:50:55.433640 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:55.433561    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:50:55.435781 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:55.435758    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ecbb10-dcd6-4827-93fc-7ffc14508a87-metrics-certs\") pod \"network-metrics-daemon-9qqr9\" (UID: \"e4ecbb10-dcd6-4827-93fc-7ffc14508a87\") " pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:50:55.592101 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:55.592070    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mqvtg\""
Jun 02 15:50:55.594754 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:55.594729    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qqr9"
Jun 02 15:50:55.735470 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:50:55.735436    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ecbb10_dcd6_4827_93fc_7ffc14508a87.slice/crio-cb6e763fba7607c0003f997eda4ba5335c039fa5905736fe49bb64f66c98ea7f WatchSource:0}: Error finding container cb6e763fba7607c0003f997eda4ba5335c039fa5905736fe49bb64f66c98ea7f: Status 404 returned error can't find the container with id cb6e763fba7607c0003f997eda4ba5335c039fa5905736fe49bb64f66c98ea7f
Jun 02 15:50:55.736029 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:55.735999    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qqr9"]
Jun 02 15:50:56.431242 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:56.431206    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qqr9" event={"ID":"e4ecbb10-dcd6-4827-93fc-7ffc14508a87","Type":"ContainerStarted","Data":"cb6e763fba7607c0003f997eda4ba5335c039fa5905736fe49bb64f66c98ea7f"}
Jun 02 15:50:57.437996 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:57.437953    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qqr9" event={"ID":"e4ecbb10-dcd6-4827-93fc-7ffc14508a87","Type":"ContainerStarted","Data":"343ac273fae4b0590b07d7085970338387c6e9ed2ca936e5d4eefeb102cf2192"}
Jun 02 15:50:57.437996 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:57.438002    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qqr9" event={"ID":"e4ecbb10-dcd6-4827-93fc-7ffc14508a87","Type":"ContainerStarted","Data":"7ce0ef231f47042af5a5efb3812389c29f11de6af7d209319fa0bc2310cb9484"}
Jun 02 15:50:57.474424 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:50:57.474353    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9qqr9" podStartSLOduration=253.443630991 podStartE2EDuration="4m14.474338503s" podCreationTimestamp="2026-06-02 15:46:43 +0000 UTC" firstStartedPulling="2026-06-02 15:50:55.737264156 +0000 UTC m=+252.612790295" lastFinishedPulling="2026-06-02 15:50:56.767971668 +0000 UTC m=+253.643497807" observedRunningTime="2026-06-02 15:50:57.468121747 +0000 UTC m=+254.343647914" watchObservedRunningTime="2026-06-02 15:50:57.474338503 +0000 UTC m=+254.349864666"
Jun 02 15:51:15.433232 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.433168    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-785644856d-rbvqj" podUID="17c23798-bc35-4951-855c-3905f42a8d0f" containerName="console" containerID="cri-o://1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88" gracePeriod=15
Jun 02 15:51:15.689651 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.689591    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-785644856d-rbvqj_17c23798-bc35-4951-855c-3905f42a8d0f/console/0.log"
Jun 02 15:51:15.689651 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.689648    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:51:15.784021 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.783983    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784033    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784067    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784102    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784137    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784191 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784177    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt56g\" (UniqueName: \"kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g\") pod \"17c23798-bc35-4951-855c-3905f42a8d0f\" (UID: \"17c23798-bc35-4951-855c-3905f42a8d0f\") "
Jun 02 15:51:15.784675 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784474    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:15.784675 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784567    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca" (OuterVolumeSpecName: "service-ca") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:15.784675 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.784580    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config" (OuterVolumeSpecName: "console-config") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:15.786408 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.786359    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:15.786685 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.786665    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:15.786755 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.786723    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g" (OuterVolumeSpecName: "kube-api-access-zt56g") pod "17c23798-bc35-4951-855c-3905f42a8d0f" (UID: "17c23798-bc35-4951-855c-3905f42a8d0f"). InnerVolumeSpecName "kube-api-access-zt56g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:51:15.885128 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885095    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-service-ca\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:15.885128 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885126    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt56g\" (UniqueName: \"kubernetes.io/projected/17c23798-bc35-4951-855c-3905f42a8d0f-kube-api-access-zt56g\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:15.885128 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885138    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-oauth-serving-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:15.885349 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885147    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c23798-bc35-4951-855c-3905f42a8d0f-console-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:15.885349 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885157    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-oauth-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:15.885349 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:15.885166    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c23798-bc35-4951-855c-3905f42a8d0f-console-serving-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:51:16.491282 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491253    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-785644856d-rbvqj_17c23798-bc35-4951-855c-3905f42a8d0f/console/0.log"
Jun 02 15:51:16.491780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491292    2571 generic.go:358] "Generic (PLEG): container finished" podID="17c23798-bc35-4951-855c-3905f42a8d0f" containerID="1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88" exitCode=2
Jun 02 15:51:16.491780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491336    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785644856d-rbvqj" event={"ID":"17c23798-bc35-4951-855c-3905f42a8d0f","Type":"ContainerDied","Data":"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"}
Jun 02 15:51:16.491780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491359    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785644856d-rbvqj" event={"ID":"17c23798-bc35-4951-855c-3905f42a8d0f","Type":"ContainerDied","Data":"875de03fcf50166d8cd84fe9fbdb49139026a07030a141fad499851b383d6bb6"}
Jun 02 15:51:16.491780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491395    2571 scope.go:117] "RemoveContainer" containerID="1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"
Jun 02 15:51:16.491780 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.491418    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785644856d-rbvqj"
Jun 02 15:51:16.501159 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.501132    2571 scope.go:117] "RemoveContainer" containerID="1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"
Jun 02 15:51:16.501550 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:51:16.501517    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88\": container with ID starting with 1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88 not found: ID does not exist" containerID="1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"
Jun 02 15:51:16.501652 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.501557    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88"} err="failed to get container status \"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88\": rpc error: code = NotFound desc = could not find container \"1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88\": container with ID starting with 1feac06f6ff98b7aff26e7d257a2de7a41af515c81187ae22d83935c86102a88 not found: ID does not exist"
Jun 02 15:51:16.519712 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.519686    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:51:16.526971 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:16.526937    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-785644856d-rbvqj"]
Jun 02 15:51:17.684293 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:17.684253    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c23798-bc35-4951-855c-3905f42a8d0f" path="/var/lib/kubelet/pods/17c23798-bc35-4951-855c-3905f42a8d0f/volumes"
Jun 02 15:51:43.583931 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:43.583903    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:51:43.584503 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:43.584206    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:51:43.587680 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:51:43.587659    2571 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:54:30.467356 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.467316    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75d476cbb8-smv6x"]
Jun 02 15:54:30.467923 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.467647    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c23798-bc35-4951-855c-3905f42a8d0f" containerName="console"
Jun 02 15:54:30.467923 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.467660    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c23798-bc35-4951-855c-3905f42a8d0f" containerName="console"
Jun 02 15:54:30.467923 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.467703    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="17c23798-bc35-4951-855c-3905f42a8d0f" containerName="console"
Jun 02 15:54:30.470037 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.470021    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.472525 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.472497    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k77vt\""
Jun 02 15:54:30.472741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.472639    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jun 02 15:54:30.472819 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.472776    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jun 02 15:54:30.472869 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.472778    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jun 02 15:54:30.472869 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.472852    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7reb286kbjc3a\""
Jun 02 15:54:30.473073 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.473060    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jun 02 15:54:30.473236 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.473223    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jun 02 15:54:30.483605 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.483573    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d476cbb8-smv6x"]
Jun 02 15:54:30.541610 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541610 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-grpc-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541728    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541801    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7q9\" (UniqueName: \"kubernetes.io/projected/b5546459-be11-4809-af8a-72d74619c686-kube-api-access-7r7q9\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.541867 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.541828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5546459-be11-4809-af8a-72d74619c686-metrics-client-ca\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.642697 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642661    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.642890 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642706    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.642890 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642739    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.642890 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642778    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-grpc-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.643045 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642904    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.643045 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.642953    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.643145 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.643118    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7q9\" (UniqueName: \"kubernetes.io/projected/b5546459-be11-4809-af8a-72d74619c686-kube-api-access-7r7q9\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.643200 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.643162    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5546459-be11-4809-af8a-72d74619c686-metrics-client-ca\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.643968 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.643929    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5546459-be11-4809-af8a-72d74619c686-metrics-client-ca\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.645428 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.645406    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.645686 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.645670    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.645741 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.645724    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.645782 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.645727    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.646150 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.646131    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.646311 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.646288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5546459-be11-4809-af8a-72d74619c686-secret-grpc-tls\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.652666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.652643    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7q9\" (UniqueName: \"kubernetes.io/projected/b5546459-be11-4809-af8a-72d74619c686-kube-api-access-7r7q9\") pod \"thanos-querier-75d476cbb8-smv6x\" (UID: \"b5546459-be11-4809-af8a-72d74619c686\") " pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.780111 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.780024    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:30.915427 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.915392    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75d476cbb8-smv6x"]
Jun 02 15:54:30.918235 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:54:30.918204    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5546459_be11_4809_af8a_72d74619c686.slice/crio-517ed5d1a2aa20a41931650c8f9bf7f19eced408eb83da25d3a929125bad221a WatchSource:0}: Error finding container 517ed5d1a2aa20a41931650c8f9bf7f19eced408eb83da25d3a929125bad221a: Status 404 returned error can't find the container with id 517ed5d1a2aa20a41931650c8f9bf7f19eced408eb83da25d3a929125bad221a
Jun 02 15:54:30.920037 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:30.920020    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:54:31.012254 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:31.012210    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"517ed5d1a2aa20a41931650c8f9bf7f19eced408eb83da25d3a929125bad221a"}
Jun 02 15:54:34.025595 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:34.025554    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"b671f3b98b55f050186b0cdb77b0fca04d1eb37b5bf8dfa12838e1442096917f"}
Jun 02 15:54:34.026009 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:34.025604    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"35cfc1f5c49e0b97ed89b9696e2f47c97d9d5cfb04105eaea137f024015ca68c"}
Jun 02 15:54:34.026009 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:34.025617    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"922eacc5e0fd3179c5dd55b300491143fa42db92adc5d71fa74534d9ac46ece8"}
Jun 02 15:54:34.026009 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:34.025632    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"51b0261d6f376dcd01ddbd07a0672a188bc39c1d94b28d285934521f00f94eb6"}
Jun 02 15:54:34.026009 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:34.025644    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"803568d2d1159be60b3c2f014acf98a2faea279d96586261e87fafe321924961"}
Jun 02 15:54:35.030268 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:35.030227    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" event={"ID":"b5546459-be11-4809-af8a-72d74619c686","Type":"ContainerStarted","Data":"a5c8b814dbc6709e47e1b2c9db43f34b4470b2327a16bf1ae7f3c5a52d843b2f"}
Jun 02 15:54:35.030675 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:35.030409    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:54:35.052135 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:35.052088    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x" podStartSLOduration=2.078270406 podStartE2EDuration="5.052072362s" podCreationTimestamp="2026-06-02 15:54:30 +0000 UTC" firstStartedPulling="2026-06-02 15:54:30.920147259 +0000 UTC m=+467.795673398" lastFinishedPulling="2026-06-02 15:54:33.893949216 +0000 UTC m=+470.769475354" observedRunningTime="2026-06-02 15:54:35.050494166 +0000 UTC m=+471.926020327" watchObservedRunningTime="2026-06-02 15:54:35.052072362 +0000 UTC m=+471.927598525"
Jun 02 15:54:41.039552 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:54:41.039521    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75d476cbb8-smv6x"
Jun 02 15:55:48.695108 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.695069    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 15:55:48.697419 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.697401    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.699743 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.699717    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 02 15:55:48.700007 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.699986    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7qwlp\""
Jun 02 15:55:48.700666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.700635    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 02 15:55:48.700779 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.700711    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 02 15:55:48.700963 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.700946    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 02 15:55:48.701501 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.701483    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 02 15:55:48.701583 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.701538    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 02 15:55:48.701814 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.701787    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 02 15:55:48.705772 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.705754    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jun 02 15:55:48.712140 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.712113    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 15:55:48.849085 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849263 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849263 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849263 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849263 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849431 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849278    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6hx\" (UniqueName: \"kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.849431 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.849300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950062 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.949988    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950062 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950022    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950062 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950040    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950062 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950062    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950379 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950099    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6hx\" (UniqueName: \"kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950379 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950119    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950379 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950136    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.950910 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950880    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.951045 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950880    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.951045 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.950893    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.951188 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.951168    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.953221 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.953197    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.953300 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.953261    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:48.958974 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:48.958954    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6hx\" (UniqueName: \"kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx\") pod \"console-8464c8d8cf-fpv49\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") " pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:49.006984 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:49.006943    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:49.141138 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:49.141102    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 15:55:49.145807 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:55:49.145780    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44b4970_e3ad_4c36_8112_abd0ad3f80bc.slice/crio-d0bd869559fdb68ae1f70532786e3186d8ce8524f741f17350c804380d7fa3b9 WatchSource:0}: Error finding container d0bd869559fdb68ae1f70532786e3186d8ce8524f741f17350c804380d7fa3b9: Status 404 returned error can't find the container with id d0bd869559fdb68ae1f70532786e3186d8ce8524f741f17350c804380d7fa3b9
Jun 02 15:55:49.241124 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:49.241088    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8464c8d8cf-fpv49" event={"ID":"f44b4970-e3ad-4c36-8112-abd0ad3f80bc","Type":"ContainerStarted","Data":"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"}
Jun 02 15:55:49.241124 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:49.241126    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8464c8d8cf-fpv49" event={"ID":"f44b4970-e3ad-4c36-8112-abd0ad3f80bc","Type":"ContainerStarted","Data":"d0bd869559fdb68ae1f70532786e3186d8ce8524f741f17350c804380d7fa3b9"}
Jun 02 15:55:49.262132 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:49.262073    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8464c8d8cf-fpv49" podStartSLOduration=1.262058274 podStartE2EDuration="1.262058274s" podCreationTimestamp="2026-06-02 15:55:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:55:49.260516403 +0000 UTC m=+546.136042576" watchObservedRunningTime="2026-06-02 15:55:49.262058274 +0000 UTC m=+546.137584433"
Jun 02 15:55:59.007750 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:59.007705    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:59.007750 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:59.007755    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:59.012679 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:59.012660    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:55:59.274522 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:55:59.274440    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 15:56:23.234581 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.234548    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5p997"]
Jun 02 15:56:23.239753 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.239630    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.243099 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.243076    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:56:23.272577 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.272543    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5p997"]
Jun 02 15:56:23.333452 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.333421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-dbus\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.333639 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.333469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bcb6f1f-892d-413e-b840-ea5186b36cf6-original-pull-secret\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.333639 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.333497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-kubelet-config\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.434247 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.434206    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-dbus\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.434460 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.434258    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bcb6f1f-892d-413e-b840-ea5186b36cf6-original-pull-secret\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.434460 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.434398    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-kubelet-config\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.434460 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.434446    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-kubelet-config\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.434584 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.434447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bcb6f1f-892d-413e-b840-ea5186b36cf6-dbus\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.436497 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.436477    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bcb6f1f-892d-413e-b840-ea5186b36cf6-original-pull-secret\") pod \"global-pull-secret-syncer-5p997\" (UID: \"3bcb6f1f-892d-413e-b840-ea5186b36cf6\") " pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.549526 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.549441    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5p997"
Jun 02 15:56:23.728626 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:23.728593    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5p997"]
Jun 02 15:56:23.738452 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:56:23.738422    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcb6f1f_892d_413e_b840_ea5186b36cf6.slice/crio-6d17b8018f8e9368dd39b7b04097b1f845b887c5541dcbc6a3d935a4a4885235 WatchSource:0}: Error finding container 6d17b8018f8e9368dd39b7b04097b1f845b887c5541dcbc6a3d935a4a4885235: Status 404 returned error can't find the container with id 6d17b8018f8e9368dd39b7b04097b1f845b887c5541dcbc6a3d935a4a4885235
Jun 02 15:56:24.344122 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:24.344088    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5p997" event={"ID":"3bcb6f1f-892d-413e-b840-ea5186b36cf6","Type":"ContainerStarted","Data":"6d17b8018f8e9368dd39b7b04097b1f845b887c5541dcbc6a3d935a4a4885235"}
Jun 02 15:56:29.359735 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:29.359693    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5p997" event={"ID":"3bcb6f1f-892d-413e-b840-ea5186b36cf6","Type":"ContainerStarted","Data":"28dd41d4f5804a2e198d2074082f73e587f22f6cb2a9398c6fc75cf26c4cab1a"}
Jun 02 15:56:29.382484 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:29.380243    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5p997" podStartSLOduration=1.58771422 podStartE2EDuration="6.380224455s" podCreationTimestamp="2026-06-02 15:56:23 +0000 UTC" firstStartedPulling="2026-06-02 15:56:23.739804415 +0000 UTC m=+580.615330555" lastFinishedPulling="2026-06-02 15:56:28.532314648 +0000 UTC m=+585.407840790" observedRunningTime="2026-06-02 15:56:29.377435684 +0000 UTC m=+586.252961845" watchObservedRunningTime="2026-06-02 15:56:29.380224455 +0000 UTC m=+586.255750618"
Jun 02 15:56:43.608423 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:43.608393    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:56:43.608842 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:56:43.608393    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 15:58:45.560477 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.560442    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"]
Jun 02 15:58:45.562977 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.562956    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.565762 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.565742    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rwhpc\""
Jun 02 15:58:45.565872 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.565743    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jun 02 15:58:45.566408 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.566392    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jun 02 15:58:45.587641 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.587614    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"]
Jun 02 15:58:45.704594 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.704560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.704594 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.704594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.704817 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.704620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q425f\" (UniqueName: \"kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.806116 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.806075    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.806116 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.806114    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.806359 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.806141    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q425f\" (UniqueName: \"kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.806572 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.806546    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.806666 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.806608    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.815130 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.815062    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q425f\" (UniqueName: \"kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:45.871753 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:45.871716    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:58:46.008379 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:46.008332    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"]
Jun 02 15:58:46.013380 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:58:46.013328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59cc7b4_2fe4_4158_890c_46015728a6b5.slice/crio-f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09 WatchSource:0}: Error finding container f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09: Status 404 returned error can't find the container with id f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09
Jun 02 15:58:46.761871 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:46.761837    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerStarted","Data":"f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09"}
Jun 02 15:58:51.780747 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:51.780706    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerStarted","Data":"9ab4807f1d69feb84d2ef229843ca046db2903f58320912871c171743e6806ea"}
Jun 02 15:58:52.784929 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:52.784894    2571 generic.go:358] "Generic (PLEG): container finished" podID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerID="9ab4807f1d69feb84d2ef229843ca046db2903f58320912871c171743e6806ea" exitCode=0
Jun 02 15:58:52.785326 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:52.784965    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerDied","Data":"9ab4807f1d69feb84d2ef229843ca046db2903f58320912871c171743e6806ea"}
Jun 02 15:58:56.797871 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:56.797833    2571 generic.go:358] "Generic (PLEG): container finished" podID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerID="160164881f931788afb4b756f058418b2fdb7cc1de8d3128c82b70b75f0eec3e" exitCode=0
Jun 02 15:58:56.798254 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:58:56.797869    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerDied","Data":"160164881f931788afb4b756f058418b2fdb7cc1de8d3128c82b70b75f0eec3e"}
Jun 02 15:59:04.824815 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:04.824780    2571 generic.go:358] "Generic (PLEG): container finished" podID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerID="498d4de60df0acb6807b5847b767f2d8338287822862fbedef9972dfa9e6d322" exitCode=0
Jun 02 15:59:04.824815 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:04.824822    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerDied","Data":"498d4de60df0acb6807b5847b767f2d8338287822862fbedef9972dfa9e6d322"}
Jun 02 15:59:05.953070 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.953047    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:59:05.976378 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.976338    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util\") pod \"f59cc7b4-2fe4-4158-890c-46015728a6b5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") "
Jun 02 15:59:05.976533 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.976410    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle\") pod \"f59cc7b4-2fe4-4158-890c-46015728a6b5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") "
Jun 02 15:59:05.976533 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.976452    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q425f\" (UniqueName: \"kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f\") pod \"f59cc7b4-2fe4-4158-890c-46015728a6b5\" (UID: \"f59cc7b4-2fe4-4158-890c-46015728a6b5\") "
Jun 02 15:59:05.976941 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.976920    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle" (OuterVolumeSpecName: "bundle") pod "f59cc7b4-2fe4-4158-890c-46015728a6b5" (UID: "f59cc7b4-2fe4-4158-890c-46015728a6b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:59:05.978791 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.978750    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f" (OuterVolumeSpecName: "kube-api-access-q425f") pod "f59cc7b4-2fe4-4158-890c-46015728a6b5" (UID: "f59cc7b4-2fe4-4158-890c-46015728a6b5"). InnerVolumeSpecName "kube-api-access-q425f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:59:05.982172 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:05.982140    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util" (OuterVolumeSpecName: "util") pod "f59cc7b4-2fe4-4158-890c-46015728a6b5" (UID: "f59cc7b4-2fe4-4158-890c-46015728a6b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:59:06.077540 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.077502    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-util\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:59:06.077540 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.077533    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f59cc7b4-2fe4-4158-890c-46015728a6b5-bundle\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:59:06.077540 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.077544    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q425f\" (UniqueName: \"kubernetes.io/projected/f59cc7b4-2fe4-4158-890c-46015728a6b5-kube-api-access-q425f\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 15:59:06.832079 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.832049    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5"
Jun 02 15:59:06.832253 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.832044    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cjk5t5" event={"ID":"f59cc7b4-2fe4-4158-890c-46015728a6b5","Type":"ContainerDied","Data":"f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09"}
Jun 02 15:59:06.832253 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:06.832153    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c342a2b9fdda99a1d47ab77121dcbfedf0313529167fca9519e773f6bf8c09"
Jun 02 15:59:12.964674 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.964637    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"]
Jun 02 15:59:12.965098 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965080    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="pull"
Jun 02 15:59:12.965140 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965101    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="pull"
Jun 02 15:59:12.965140 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965135    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="extract"
Jun 02 15:59:12.965205 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965144    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="extract"
Jun 02 15:59:12.965205 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965154    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="util"
Jun 02 15:59:12.965205 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965162    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="util"
Jun 02 15:59:12.965291 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.965225    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f59cc7b4-2fe4-4158-890c-46015728a6b5" containerName="extract"
Jun 02 15:59:12.969516 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.969496    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:12.973095 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.973075    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8fswk\""
Jun 02 15:59:12.973346 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.973333    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jun 02 15:59:12.973653 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.973640    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jun 02 15:59:12.973726 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.973711    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jun 02 15:59:12.986566 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:12.986542    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"]
Jun 02 15:59:13.031659 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.031625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.031828 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.031687    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd4w\" (UniqueName: \"kubernetes.io/projected/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-kube-api-access-wmd4w\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.132292 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.132257    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.132476 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.132323    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd4w\" (UniqueName: \"kubernetes.io/projected/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-kube-api-access-wmd4w\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.134568 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.134533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.142345 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.142321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd4w\" (UniqueName: \"kubernetes.io/projected/4ece3489-aa4a-49a5-9ec9-731c0f5be9af-kube-api-access-wmd4w\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm\" (UID: \"4ece3489-aa4a-49a5-9ec9-731c0f5be9af\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.280427 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.280320    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:13.428331 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.428301    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"]
Jun 02 15:59:13.430044 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:59:13.430017    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ece3489_aa4a_49a5_9ec9_731c0f5be9af.slice/crio-142e8713a80cff9e3ddcc3085dd19547f6e4feb3107c4897bf74676f151188c4 WatchSource:0}: Error finding container 142e8713a80cff9e3ddcc3085dd19547f6e4feb3107c4897bf74676f151188c4: Status 404 returned error can't find the container with id 142e8713a80cff9e3ddcc3085dd19547f6e4feb3107c4897bf74676f151188c4
Jun 02 15:59:13.853490 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:13.853455    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm" event={"ID":"4ece3489-aa4a-49a5-9ec9-731c0f5be9af","Type":"ContainerStarted","Data":"142e8713a80cff9e3ddcc3085dd19547f6e4feb3107c4897bf74676f151188c4"}
Jun 02 15:59:26.505212 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.505173    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-sw59g"]
Jun 02 15:59:26.508592 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.508576    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.516624 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.516606    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jun 02 15:59:26.516840 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.516827    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5rhjx\""
Jun 02 15:59:26.516903 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.516885    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jun 02 15:59:26.547865 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.547830    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-sw59g"]
Jun 02 15:59:26.550402 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.550382    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.550487 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.550454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7clx\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-kube-api-access-p7clx\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.550548 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.550534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aeddc291-98db-48a5-b50a-9a6863ad1cd4-cabundle0\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.651454 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.651420    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7clx\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-kube-api-access-p7clx\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.651478    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aeddc291-98db-48a5-b50a-9a6863ad1cd4-cabundle0\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.651532    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:26.651622    2571 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:26.651636    2571 secret.go:281] references non-existent secret key: ca.crt
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:26.651643    2571 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 02 15:59:26.651654 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:26.651656    2571 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-sw59g: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jun 02 15:59:26.652014 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:26.651702    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates podName:aeddc291-98db-48a5-b50a-9a6863ad1cd4 nodeName:}" failed. No retries permitted until 2026-06-02 15:59:27.151687913 +0000 UTC m=+764.027214053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates") pod "keda-operator-ffbb595cb-sw59g" (UID: "aeddc291-98db-48a5-b50a-9a6863ad1cd4") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jun 02 15:59:26.652218 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.652194    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/aeddc291-98db-48a5-b50a-9a6863ad1cd4-cabundle0\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.664498 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.664469    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7clx\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-kube-api-access-p7clx\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:26.897647 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.897557    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm" event={"ID":"4ece3489-aa4a-49a5-9ec9-731c0f5be9af","Type":"ContainerStarted","Data":"bf9bc06148229f66fc38a871a096bd141e2648272b2ff14ae8fadf04fcc03c6e"}
Jun 02 15:59:26.897647 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.897614    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 15:59:26.938470 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:26.938421    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm" podStartSLOduration=2.470641668 podStartE2EDuration="14.938406963s" podCreationTimestamp="2026-06-02 15:59:12 +0000 UTC" firstStartedPulling="2026-06-02 15:59:13.431806642 +0000 UTC m=+750.307332781" lastFinishedPulling="2026-06-02 15:59:25.899571936 +0000 UTC m=+762.775098076" observedRunningTime="2026-06-02 15:59:26.937028726 +0000 UTC m=+763.812554886" watchObservedRunningTime="2026-06-02 15:59:26.938406963 +0000 UTC m=+763.813933127"
Jun 02 15:59:27.154699 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:27.154604    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:27.154860 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:27.154709    2571 secret.go:281] references non-existent secret key: ca.crt
Jun 02 15:59:27.154860 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:27.154731    2571 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 02 15:59:27.154860 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:27.154740    2571 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-sw59g: references non-existent secret key: ca.crt
Jun 02 15:59:27.154860 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:27.154798    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates podName:aeddc291-98db-48a5-b50a-9a6863ad1cd4 nodeName:}" failed. No retries permitted until 2026-06-02 15:59:28.154783802 +0000 UTC m=+765.030309940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates") pod "keda-operator-ffbb595cb-sw59g" (UID: "aeddc291-98db-48a5-b50a-9a6863ad1cd4") : references non-existent secret key: ca.crt
Jun 02 15:59:28.165515 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:28.165479    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:28.165901 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:28.165624    2571 secret.go:281] references non-existent secret key: ca.crt
Jun 02 15:59:28.165901 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:28.165639    2571 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 02 15:59:28.165901 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:28.165649    2571 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-sw59g: references non-existent secret key: ca.crt
Jun 02 15:59:28.165901 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:28.165708    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates podName:aeddc291-98db-48a5-b50a-9a6863ad1cd4 nodeName:}" failed. No retries permitted until 2026-06-02 15:59:30.165689497 +0000 UTC m=+767.041215653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates") pod "keda-operator-ffbb595cb-sw59g" (UID: "aeddc291-98db-48a5-b50a-9a6863ad1cd4") : references non-existent secret key: ca.crt
Jun 02 15:59:30.181305 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:30.181257    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:30.181830 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:30.181450    2571 secret.go:281] references non-existent secret key: ca.crt
Jun 02 15:59:30.181830 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:30.181477    2571 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 02 15:59:30.181830 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:30.181494    2571 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-sw59g: references non-existent secret key: ca.crt
Jun 02 15:59:30.181830 ip-10-0-131-30 kubenswrapper[2571]: E0602 15:59:30.181565    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates podName:aeddc291-98db-48a5-b50a-9a6863ad1cd4 nodeName:}" failed. No retries permitted until 2026-06-02 15:59:34.181545167 +0000 UTC m=+771.057071305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates") pod "keda-operator-ffbb595cb-sw59g" (UID: "aeddc291-98db-48a5-b50a-9a6863ad1cd4") : references non-existent secret key: ca.crt
Jun 02 15:59:34.213475 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.213432    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:34.215923 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.215901    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aeddc291-98db-48a5-b50a-9a6863ad1cd4-certificates\") pod \"keda-operator-ffbb595cb-sw59g\" (UID: \"aeddc291-98db-48a5-b50a-9a6863ad1cd4\") " pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:34.319118 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.319069    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:34.452050 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.452019    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-sw59g"]
Jun 02 15:59:34.466817 ip-10-0-131-30 kubenswrapper[2571]: W0602 15:59:34.466781    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeddc291_98db_48a5_b50a_9a6863ad1cd4.slice/crio-def4065c312dc30542c3af9770bc323084e23e0fd694bdd44419a717d6a539a2 WatchSource:0}: Error finding container def4065c312dc30542c3af9770bc323084e23e0fd694bdd44419a717d6a539a2: Status 404 returned error can't find the container with id def4065c312dc30542c3af9770bc323084e23e0fd694bdd44419a717d6a539a2
Jun 02 15:59:34.468150 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.468130    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:59:34.923420 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:34.923320    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-sw59g" event={"ID":"aeddc291-98db-48a5-b50a-9a6863ad1cd4","Type":"ContainerStarted","Data":"def4065c312dc30542c3af9770bc323084e23e0fd694bdd44419a717d6a539a2"}
Jun 02 15:59:39.941498 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:39.941457    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-sw59g" event={"ID":"aeddc291-98db-48a5-b50a-9a6863ad1cd4","Type":"ContainerStarted","Data":"2c850fc623ebc11e625c5620ea761186d0050ab101addacc3e9a74dda239cced"}
Jun 02 15:59:39.941896 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:39.941580    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 15:59:39.976941 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:39.976884    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-sw59g" podStartSLOduration=9.148098203 podStartE2EDuration="13.976868222s" podCreationTimestamp="2026-06-02 15:59:26 +0000 UTC" firstStartedPulling="2026-06-02 15:59:34.468319855 +0000 UTC m=+771.343845995" lastFinishedPulling="2026-06-02 15:59:39.297089869 +0000 UTC m=+776.172616014" observedRunningTime="2026-06-02 15:59:39.97572297 +0000 UTC m=+776.851249133" watchObservedRunningTime="2026-06-02 15:59:39.976868222 +0000 UTC m=+776.852394428"
Jun 02 15:59:47.903522 ip-10-0-131-30 kubenswrapper[2571]: I0602 15:59:47.903482    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-vv7nm"
Jun 02 16:00:00.946211 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:00.946138    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-sw59g"
Jun 02 16:00:35.769592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.769557    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:00:35.772118 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.772097    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:35.782980 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.782957    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 02 16:00:35.783101 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.782960    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jun 02 16:00:35.784203 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.784188    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 02 16:00:35.784269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.784194    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-x67l7\""
Jun 02 16:00:35.808539 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.808513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:35.808766 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.808744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5rjp\" (UniqueName: \"kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:35.811534 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.811507    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:00:35.813317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.813297    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"]
Jun 02 16:00:35.816016 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.815991    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:35.819624 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.819592    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jun 02 16:00:35.819730 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.819714    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-m2kdv\""
Jun 02 16:00:35.837952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.837918    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"]
Jun 02 16:00:35.910151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.910113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:35.910318 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.910186    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:35.910318 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.910222    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q5rjp\" (UniqueName: \"kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:35.910318 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.910272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6s9\" (UniqueName: \"kubernetes.io/projected/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-kube-api-access-hw6s9\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:35.910482 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:00:35.910322    2571 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jun 02 16:00:35.910482 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:00:35.910406    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert podName:6b24e207-326c-4998-a6a0-3be8091ecff7 nodeName:}" failed. No retries permitted until 2026-06-02 16:00:36.410388164 +0000 UTC m=+833.285914309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert") pod "kserve-controller-manager-7b649f477f-r4pfj" (UID: "6b24e207-326c-4998-a6a0-3be8091ecff7") : secret "kserve-webhook-server-cert" not found
Jun 02 16:00:35.938106 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:35.938058    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5rjp\" (UniqueName: \"kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:36.010952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.010919    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6s9\" (UniqueName: \"kubernetes.io/projected/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-kube-api-access-hw6s9\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:36.010952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.010961    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:36.013323 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.013303    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:36.020962 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.020900    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6s9\" (UniqueName: \"kubernetes.io/projected/140e5bfd-cad1-4b86-935f-0baa9fd73d4f-kube-api-access-hw6s9\") pod \"llmisvc-controller-manager-6b94ff949c-9fj7n\" (UID: \"140e5bfd-cad1-4b86-935f-0baa9fd73d4f\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:36.125492 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.125464    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:36.267004 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.266982    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"]
Jun 02 16:00:36.269583 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:00:36.269556    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod140e5bfd_cad1_4b86_935f_0baa9fd73d4f.slice/crio-e1b711ad6116e69ec744c1e727f9bdb2f85ec3c2cce0a1e3fe32a7eebcececb5 WatchSource:0}: Error finding container e1b711ad6116e69ec744c1e727f9bdb2f85ec3c2cce0a1e3fe32a7eebcececb5: Status 404 returned error can't find the container with id e1b711ad6116e69ec744c1e727f9bdb2f85ec3c2cce0a1e3fe32a7eebcececb5
Jun 02 16:00:36.414674 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.414636    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:36.417013 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.416983    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") pod \"kserve-controller-manager-7b649f477f-r4pfj\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") " pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:36.682317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.682236    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:36.827980 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:36.827946    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:00:36.832341 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:00:36.832314    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b24e207_326c_4998_a6a0_3be8091ecff7.slice/crio-9ad5980ea314fb6ff205d036b1dbf21a12936c563d9c27dfe9940b780fa0e21f WatchSource:0}: Error finding container 9ad5980ea314fb6ff205d036b1dbf21a12936c563d9c27dfe9940b780fa0e21f: Status 404 returned error can't find the container with id 9ad5980ea314fb6ff205d036b1dbf21a12936c563d9c27dfe9940b780fa0e21f
Jun 02 16:00:37.130327 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:37.130296    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" event={"ID":"6b24e207-326c-4998-a6a0-3be8091ecff7","Type":"ContainerStarted","Data":"9ad5980ea314fb6ff205d036b1dbf21a12936c563d9c27dfe9940b780fa0e21f"}
Jun 02 16:00:37.131408 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:37.131361    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n" event={"ID":"140e5bfd-cad1-4b86-935f-0baa9fd73d4f","Type":"ContainerStarted","Data":"e1b711ad6116e69ec744c1e727f9bdb2f85ec3c2cce0a1e3fe32a7eebcececb5"}
Jun 02 16:00:40.152556 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.152498    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n" event={"ID":"140e5bfd-cad1-4b86-935f-0baa9fd73d4f","Type":"ContainerStarted","Data":"348670fd5f410cf4799117af3541af6de7a3ac943846462f2d5c6cc77d02fa5b"}
Jun 02 16:00:40.152973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.152595    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:00:40.153943 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.153914    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" event={"ID":"6b24e207-326c-4998-a6a0-3be8091ecff7","Type":"ContainerStarted","Data":"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"}
Jun 02 16:00:40.154075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.154060    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:00:40.193513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.193459    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n" podStartSLOduration=1.400151293 podStartE2EDuration="5.193444151s" podCreationTimestamp="2026-06-02 16:00:35 +0000 UTC" firstStartedPulling="2026-06-02 16:00:36.270957664 +0000 UTC m=+833.146483803" lastFinishedPulling="2026-06-02 16:00:40.064250522 +0000 UTC m=+836.939776661" observedRunningTime="2026-06-02 16:00:40.192432347 +0000 UTC m=+837.067958510" watchObservedRunningTime="2026-06-02 16:00:40.193444151 +0000 UTC m=+837.068970312"
Jun 02 16:00:40.235094 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:40.235044    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" podStartSLOduration=1.999765483 podStartE2EDuration="5.235029052s" podCreationTimestamp="2026-06-02 16:00:35 +0000 UTC" firstStartedPulling="2026-06-02 16:00:36.833664336 +0000 UTC m=+833.709190475" lastFinishedPulling="2026-06-02 16:00:40.068927905 +0000 UTC m=+836.944454044" observedRunningTime="2026-06-02 16:00:40.232345104 +0000 UTC m=+837.107871265" watchObservedRunningTime="2026-06-02 16:00:40.235029052 +0000 UTC m=+837.110555213"
Jun 02 16:00:49.475721 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.475676    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-76fc654ccf-xnzsv"]
Jun 02 16:00:49.478204 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.478183    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.493423 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.493362    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fc654ccf-xnzsv"]
Jun 02 16:00:49.525379 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-console-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525545 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-service-ca\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525545 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525500    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-trusted-ca-bundle\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525545 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-oauth-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4b7\" (UniqueName: \"kubernetes.io/projected/75e12a91-4ab5-4212-8307-69a891bc4b34-kube-api-access-mr4b7\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-oauth-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.525652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.525644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.626805 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626755    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.626805 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626810    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-console-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627040 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626840    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-service-ca\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627040 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626873    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-trusted-ca-bundle\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627040 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626894    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-oauth-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627040 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626935    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4b7\" (UniqueName: \"kubernetes.io/projected/75e12a91-4ab5-4212-8307-69a891bc4b34-kube-api-access-mr4b7\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627040 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.626976    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-oauth-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627695 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.627664    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-console-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627829 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.627690    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-oauth-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627829 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.627717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-trusted-ca-bundle\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.627829 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.627769    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75e12a91-4ab5-4212-8307-69a891bc4b34-service-ca\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.629355 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.629335    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-serving-cert\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.629463 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.629360    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75e12a91-4ab5-4212-8307-69a891bc4b34-console-oauth-config\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.635856 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.635833    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4b7\" (UniqueName: \"kubernetes.io/projected/75e12a91-4ab5-4212-8307-69a891bc4b34-kube-api-access-mr4b7\") pod \"console-76fc654ccf-xnzsv\" (UID: \"75e12a91-4ab5-4212-8307-69a891bc4b34\") " pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.793821 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.793738    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:49.924130 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:49.924097    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fc654ccf-xnzsv"]
Jun 02 16:00:49.926740 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:00:49.926710    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e12a91_4ab5_4212_8307_69a891bc4b34.slice/crio-1c54fa4708a1bd7d8d52e9318b9fb17d6bc63d95ef0a3e58de59f6720e4b2ea8 WatchSource:0}: Error finding container 1c54fa4708a1bd7d8d52e9318b9fb17d6bc63d95ef0a3e58de59f6720e4b2ea8: Status 404 returned error can't find the container with id 1c54fa4708a1bd7d8d52e9318b9fb17d6bc63d95ef0a3e58de59f6720e4b2ea8
Jun 02 16:00:50.188450 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:50.188318    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc654ccf-xnzsv" event={"ID":"75e12a91-4ab5-4212-8307-69a891bc4b34","Type":"ContainerStarted","Data":"feade320902932567830f59ea18c4af1e969c25d57f066ff157ff332ba1e667b"}
Jun 02 16:00:50.188450 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:50.188361    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc654ccf-xnzsv" event={"ID":"75e12a91-4ab5-4212-8307-69a891bc4b34","Type":"ContainerStarted","Data":"1c54fa4708a1bd7d8d52e9318b9fb17d6bc63d95ef0a3e58de59f6720e4b2ea8"}
Jun 02 16:00:59.794595 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:59.794556    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:59.794595 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:59.794607    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:59.800168 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:59.800143    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:00:59.821391 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:00:59.821313    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-76fc654ccf-xnzsv" podStartSLOduration=10.821294144 podStartE2EDuration="10.821294144s" podCreationTimestamp="2026-06-02 16:00:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:00:50.21882397 +0000 UTC m=+847.094350131" watchObservedRunningTime="2026-06-02 16:00:59.821294144 +0000 UTC m=+856.696820306"
Jun 02 16:01:00.225827 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:00.225799    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76fc654ccf-xnzsv"
Jun 02 16:01:00.293213 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:00.293182    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 16:01:11.160229 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:11.160183    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-9fj7n"
Jun 02 16:01:11.163254 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:11.163228    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:01:12.762505 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.762468    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:01:12.762967 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.762685    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" podUID="6b24e207-326c-4998-a6a0-3be8091ecff7" containerName="manager" containerID="cri-o://9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174" gracePeriod=10
Jun 02 16:01:12.807606 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.807570    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-8jx6t"]
Jun 02 16:01:12.843249 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.843219    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-8jx6t"]
Jun 02 16:01:12.843413 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.843393    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:12.926549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.926515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6fde234-c306-4280-95a5-f41746e4665a-cert\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:12.926735 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:12.926554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4lf\" (UniqueName: \"kubernetes.io/projected/f6fde234-c306-4280-95a5-f41746e4665a-kube-api-access-lj4lf\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.019599 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.019539    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:01:13.027180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.027158    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6fde234-c306-4280-95a5-f41746e4665a-cert\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.027308 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.027187    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4lf\" (UniqueName: \"kubernetes.io/projected/f6fde234-c306-4280-95a5-f41746e4665a-kube-api-access-lj4lf\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.029550 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.029525    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6fde234-c306-4280-95a5-f41746e4665a-cert\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.037279 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.037260    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4lf\" (UniqueName: \"kubernetes.io/projected/f6fde234-c306-4280-95a5-f41746e4665a-kube-api-access-lj4lf\") pod \"kserve-controller-manager-7b649f477f-8jx6t\" (UID: \"f6fde234-c306-4280-95a5-f41746e4665a\") " pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.127663 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.127627    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") pod \"6b24e207-326c-4998-a6a0-3be8091ecff7\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") "
Jun 02 16:01:13.127839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.127708    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5rjp\" (UniqueName: \"kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp\") pod \"6b24e207-326c-4998-a6a0-3be8091ecff7\" (UID: \"6b24e207-326c-4998-a6a0-3be8091ecff7\") "
Jun 02 16:01:13.129846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.129810    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert" (OuterVolumeSpecName: "cert") pod "6b24e207-326c-4998-a6a0-3be8091ecff7" (UID: "6b24e207-326c-4998-a6a0-3be8091ecff7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:13.129958 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.129878    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp" (OuterVolumeSpecName: "kube-api-access-q5rjp") pod "6b24e207-326c-4998-a6a0-3be8091ecff7" (UID: "6b24e207-326c-4998-a6a0-3be8091ecff7"). InnerVolumeSpecName "kube-api-access-q5rjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:13.209101 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.209061    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:13.231165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.231136    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5rjp\" (UniqueName: \"kubernetes.io/projected/6b24e207-326c-4998-a6a0-3be8091ecff7-kube-api-access-q5rjp\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:13.231165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.231166    2571 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b24e207-326c-4998-a6a0-3be8091ecff7-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:13.268223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.268179    2571 generic.go:358] "Generic (PLEG): container finished" podID="6b24e207-326c-4998-a6a0-3be8091ecff7" containerID="9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174" exitCode=0
Jun 02 16:01:13.268409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.268300    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" event={"ID":"6b24e207-326c-4998-a6a0-3be8091ecff7","Type":"ContainerDied","Data":"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"}
Jun 02 16:01:13.268409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.268337    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj" event={"ID":"6b24e207-326c-4998-a6a0-3be8091ecff7","Type":"ContainerDied","Data":"9ad5980ea314fb6ff205d036b1dbf21a12936c563d9c27dfe9940b780fa0e21f"}
Jun 02 16:01:13.268409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.268361    2571 scope.go:117] "RemoveContainer" containerID="9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"
Jun 02 16:01:13.268626 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.268577    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b649f477f-r4pfj"
Jun 02 16:01:13.278101 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.278069    2571 scope.go:117] "RemoveContainer" containerID="9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"
Jun 02 16:01:13.278428 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:01:13.278405    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174\": container with ID starting with 9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174 not found: ID does not exist" containerID="9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"
Jun 02 16:01:13.278530 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.278437    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174"} err="failed to get container status \"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174\": rpc error: code = NotFound desc = could not find container \"9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174\": container with ID starting with 9225557527a6138440d389c078d2c19f0fb7b8ad92d3fb974d50a19bc9e7e174 not found: ID does not exist"
Jun 02 16:01:13.299786 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.299756    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:01:13.306328 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.306305    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-r4pfj"]
Jun 02 16:01:13.348954 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.348921    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b649f477f-8jx6t"]
Jun 02 16:01:13.352485 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:01:13.352460    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fde234_c306_4280_95a5_f41746e4665a.slice/crio-c5f99d758889c8ec50d69802fa4d41c99f6f13400119e553c615b32582d6adf4 WatchSource:0}: Error finding container c5f99d758889c8ec50d69802fa4d41c99f6f13400119e553c615b32582d6adf4: Status 404 returned error can't find the container with id c5f99d758889c8ec50d69802fa4d41c99f6f13400119e553c615b32582d6adf4
Jun 02 16:01:13.683630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:13.683552    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b24e207-326c-4998-a6a0-3be8091ecff7" path="/var/lib/kubelet/pods/6b24e207-326c-4998-a6a0-3be8091ecff7/volumes"
Jun 02 16:01:14.276195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:14.276163    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t" event={"ID":"f6fde234-c306-4280-95a5-f41746e4665a","Type":"ContainerStarted","Data":"74227c745b80a970829a6b8e86d94c9381404ae75c86d180b33541340f68c1cb"}
Jun 02 16:01:14.276195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:14.276197    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t" event={"ID":"f6fde234-c306-4280-95a5-f41746e4665a","Type":"ContainerStarted","Data":"c5f99d758889c8ec50d69802fa4d41c99f6f13400119e553c615b32582d6adf4"}
Jun 02 16:01:14.276629 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:14.276228    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:14.311838 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:14.311782    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t" podStartSLOduration=1.75643053 podStartE2EDuration="2.311769341s" podCreationTimestamp="2026-06-02 16:01:12 +0000 UTC" firstStartedPulling="2026-06-02 16:01:13.353753472 +0000 UTC m=+870.229279612" lastFinishedPulling="2026-06-02 16:01:13.909092282 +0000 UTC m=+870.784618423" observedRunningTime="2026-06-02 16:01:14.308340946 +0000 UTC m=+871.183867107" watchObservedRunningTime="2026-06-02 16:01:14.311769341 +0000 UTC m=+871.187295568"
Jun 02 16:01:25.314817 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.314775    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8464c8d8cf-fpv49" podUID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" containerName="console" containerID="cri-o://505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842" gracePeriod=15
Jun 02 16:01:25.558905 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.558881    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8464c8d8cf-fpv49_f44b4970-e3ad-4c36-8112-abd0ad3f80bc/console/0.log"
Jun 02 16:01:25.559028 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.558954    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 16:01:25.635878 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.635792    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.635878 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.635843    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.635878 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.635871    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.636098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.635904    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm6hx\" (UniqueName: \"kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.636098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.635976    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.636098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636000    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.636098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636041    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca\") pod \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\" (UID: \"f44b4970-e3ad-4c36-8112-abd0ad3f80bc\") "
Jun 02 16:01:25.636341 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636309    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config" (OuterVolumeSpecName: "console-config") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:25.636437 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636347    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:25.636437 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636396    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:25.636664 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.636644    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:01:25.638248 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.638213    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx" (OuterVolumeSpecName: "kube-api-access-mm6hx") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "kube-api-access-mm6hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:25.638248 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.638225    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:25.638405 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.638284    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f44b4970-e3ad-4c36-8112-abd0ad3f80bc" (UID: "f44b4970-e3ad-4c36-8112-abd0ad3f80bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:25.737512 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737475    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-serving-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737512 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737507    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-trusted-ca-bundle\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737512 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737518    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737528    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6hx\" (UniqueName: \"kubernetes.io/projected/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-kube-api-access-mm6hx\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737537    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-oauth-serving-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737546    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-console-oauth-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:25.737739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:25.737554    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f44b4970-e3ad-4c36-8112-abd0ad3f80bc-service-ca\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:01:26.317678 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317653    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8464c8d8cf-fpv49_f44b4970-e3ad-4c36-8112-abd0ad3f80bc/console/0.log"
Jun 02 16:01:26.318133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317691    2571 generic.go:358] "Generic (PLEG): container finished" podID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" containerID="505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842" exitCode=2
Jun 02 16:01:26.318133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317770    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8464c8d8cf-fpv49" event={"ID":"f44b4970-e3ad-4c36-8112-abd0ad3f80bc","Type":"ContainerDied","Data":"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"}
Jun 02 16:01:26.318133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317795    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8464c8d8cf-fpv49" event={"ID":"f44b4970-e3ad-4c36-8112-abd0ad3f80bc","Type":"ContainerDied","Data":"d0bd869559fdb68ae1f70532786e3186d8ce8524f741f17350c804380d7fa3b9"}
Jun 02 16:01:26.318133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317810    2571 scope.go:117] "RemoveContainer" containerID="505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"
Jun 02 16:01:26.318133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.317812    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8464c8d8cf-fpv49"
Jun 02 16:01:26.326077 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.326056    2571 scope.go:117] "RemoveContainer" containerID="505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"
Jun 02 16:01:26.326343 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:01:26.326321    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842\": container with ID starting with 505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842 not found: ID does not exist" containerID="505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"
Jun 02 16:01:26.326466 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.326356    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842"} err="failed to get container status \"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842\": rpc error: code = NotFound desc = could not find container \"505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842\": container with ID starting with 505d26cb4fdf98c6b2d312cd1d51ea8bd5d2b59e6bb0b2ad7bd6e0043a0c0842 not found: ID does not exist"
Jun 02 16:01:26.339477 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.339448    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 16:01:26.342481 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:26.342460    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8464c8d8cf-fpv49"]
Jun 02 16:01:27.685121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:27.685089    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" path="/var/lib/kubelet/pods/f44b4970-e3ad-4c36-8112-abd0ad3f80bc/volumes"
Jun 02 16:01:43.631405 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:43.631358    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:01:43.631885 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:43.631483    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:01:45.283190 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:45.283159    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b649f477f-8jx6t"
Jun 02 16:01:46.243836 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.243800    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-ggjmn"]
Jun 02 16:01:46.244136 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244124    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" containerName="console"
Jun 02 16:01:46.244180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244138    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" containerName="console"
Jun 02 16:01:46.244180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244160    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b24e207-326c-4998-a6a0-3be8091ecff7" containerName="manager"
Jun 02 16:01:46.244180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244176    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b24e207-326c-4998-a6a0-3be8091ecff7" containerName="manager"
Jun 02 16:01:46.244274 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244226    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6b24e207-326c-4998-a6a0-3be8091ecff7" containerName="manager"
Jun 02 16:01:46.244274 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.244239    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f44b4970-e3ad-4c36-8112-abd0ad3f80bc" containerName="console"
Jun 02 16:01:46.247614 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.247585    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.249949 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.249929    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jun 02 16:01:46.250078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.249962    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-w7m2b\""
Jun 02 16:01:46.258199 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.258176    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-ggjmn"]
Jun 02 16:01:46.308080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.308046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d45046-1219-4117-a3b5-19b3c7d8f86d-tls-certs\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.308475 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.308095    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz7x\" (UniqueName: \"kubernetes.io/projected/41d45046-1219-4117-a3b5-19b3c7d8f86d-kube-api-access-pwz7x\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.409113 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.409075    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d45046-1219-4117-a3b5-19b3c7d8f86d-tls-certs\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.409113 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.409128    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz7x\" (UniqueName: \"kubernetes.io/projected/41d45046-1219-4117-a3b5-19b3c7d8f86d-kube-api-access-pwz7x\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.411595 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.411567    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d45046-1219-4117-a3b5-19b3c7d8f86d-tls-certs\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.417977 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.417950    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz7x\" (UniqueName: \"kubernetes.io/projected/41d45046-1219-4117-a3b5-19b3c7d8f86d-kube-api-access-pwz7x\") pod \"model-serving-api-86f7b4b499-ggjmn\" (UID: \"41d45046-1219-4117-a3b5-19b3c7d8f86d\") " pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.557910 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.557823    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:46.709330 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:01:46.709298    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d45046_1219_4117_a3b5_19b3c7d8f86d.slice/crio-d4f42ce646859ac4bcb0a1fde26ef42df2a8770646f29f6ae019573f7c5007fa WatchSource:0}: Error finding container d4f42ce646859ac4bcb0a1fde26ef42df2a8770646f29f6ae019573f7c5007fa: Status 404 returned error can't find the container with id d4f42ce646859ac4bcb0a1fde26ef42df2a8770646f29f6ae019573f7c5007fa
Jun 02 16:01:46.712900 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:46.712875    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-ggjmn"]
Jun 02 16:01:47.388602 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:47.388561    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-ggjmn" event={"ID":"41d45046-1219-4117-a3b5-19b3c7d8f86d","Type":"ContainerStarted","Data":"d4f42ce646859ac4bcb0a1fde26ef42df2a8770646f29f6ae019573f7c5007fa"}
Jun 02 16:01:49.397470 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:49.397438    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-ggjmn" event={"ID":"41d45046-1219-4117-a3b5-19b3c7d8f86d","Type":"ContainerStarted","Data":"859048b5572fbc57d2945deeeb0d8e739a710ba1114941a4cc3d11130518e0c8"}
Jun 02 16:01:49.397846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:49.397632    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:01:49.417274 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:01:49.417188    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-ggjmn" podStartSLOduration=1.237300021 podStartE2EDuration="3.417168336s" podCreationTimestamp="2026-06-02 16:01:46 +0000 UTC" firstStartedPulling="2026-06-02 16:01:46.711813386 +0000 UTC m=+903.587339526" lastFinishedPulling="2026-06-02 16:01:48.891681702 +0000 UTC m=+905.767207841" observedRunningTime="2026-06-02 16:01:49.41553722 +0000 UTC m=+906.291063373" watchObservedRunningTime="2026-06-02 16:01:49.417168336 +0000 UTC m=+906.292694499"
Jun 02 16:02:00.405168 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:00.405137    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-ggjmn"
Jun 02 16:02:18.794890 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.794854    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"]
Jun 02 16:02:18.797084 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.797068    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.803609 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.803489    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jun 02 16:02:18.804210 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.804190    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jun 02 16:02:18.804314 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.804262    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lqfnk\""
Jun 02 16:02:18.813451 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.813425    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"]
Jun 02 16:02:18.888430 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.888387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rbl\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-kube-api-access-w9rbl\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.888634 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.888515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.888634 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.888564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.989344 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.989303    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rbl\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-kube-api-access-w9rbl\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.989561 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.989360    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.989561 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.989411    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.989761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.989741    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.991899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.991867    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:18.997543 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:18.997516    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rbl\" (UniqueName: \"kubernetes.io/projected/c3f163e2-c7b9-4a83-a06c-fd581f9b0733-kube-api-access-w9rbl\") pod \"seaweedfs-tls-custom-5c88b85bb7-lrj4v\" (UID: \"c3f163e2-c7b9-4a83-a06c-fd581f9b0733\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:19.106822 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:19.106726    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"
Jun 02 16:02:19.237566 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:19.237532    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v"]
Jun 02 16:02:19.240676 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:02:19.240647    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f163e2_c7b9_4a83_a06c_fd581f9b0733.slice/crio-5424664b62d1d7df54be4ce5c420456024ad796d96c24df708e7425f1d8cba98 WatchSource:0}: Error finding container 5424664b62d1d7df54be4ce5c420456024ad796d96c24df708e7425f1d8cba98: Status 404 returned error can't find the container with id 5424664b62d1d7df54be4ce5c420456024ad796d96c24df708e7425f1d8cba98
Jun 02 16:02:19.496876 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:19.496837    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v" event={"ID":"c3f163e2-c7b9-4a83-a06c-fd581f9b0733","Type":"ContainerStarted","Data":"5424664b62d1d7df54be4ce5c420456024ad796d96c24df708e7425f1d8cba98"}
Jun 02 16:02:22.509442 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:22.509403    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v" event={"ID":"c3f163e2-c7b9-4a83-a06c-fd581f9b0733","Type":"ContainerStarted","Data":"be583702c54e4bd069a024f1db8838faa456e4d8458b51e59380b8a8afdd5859"}
Jun 02 16:02:22.525877 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:22.525828    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-lrj4v" podStartSLOduration=1.87622308 podStartE2EDuration="4.525813707s" podCreationTimestamp="2026-06-02 16:02:18 +0000 UTC" firstStartedPulling="2026-06-02 16:02:19.241929144 +0000 UTC m=+936.117455287" lastFinishedPulling="2026-06-02 16:02:21.891519776 +0000 UTC m=+938.767045914" observedRunningTime="2026-06-02 16:02:22.524184294 +0000 UTC m=+939.399710456" watchObservedRunningTime="2026-06-02 16:02:22.525813707 +0000 UTC m=+939.401339871"
Jun 02 16:02:31.298904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.298861    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"]
Jun 02 16:02:31.302080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.302061    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.304737 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.304714    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jun 02 16:02:31.304877 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.304769    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jun 02 16:02:31.312166 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.312137    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"]
Jun 02 16:02:31.388637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.388595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.388825 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.388663    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/345ef462-6ddc-498f-a6ff-0c33412fd7a3-data\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.388825 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.388694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ptn\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-kube-api-access-57ptn\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.489196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.489158    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57ptn\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-kube-api-access-57ptn\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.489400 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.489220    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.489400 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.489270    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/345ef462-6ddc-498f-a6ff-0c33412fd7a3-data\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.489639 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.489620    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/345ef462-6ddc-498f-a6ff-0c33412fd7a3-data\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.491699 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.491676    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.499005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.498985    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ptn\" (UniqueName: \"kubernetes.io/projected/345ef462-6ddc-498f-a6ff-0c33412fd7a3-kube-api-access-57ptn\") pod \"seaweedfs-tls-serving-7fd5766db9-4l4xg\" (UID: \"345ef462-6ddc-498f-a6ff-0c33412fd7a3\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.612053 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.611965    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"
Jun 02 16:02:31.742758 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:31.742730    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg"]
Jun 02 16:02:31.744851 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:02:31.744823    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345ef462_6ddc_498f_a6ff_0c33412fd7a3.slice/crio-ae0c14ab0675025b6798aefc2e90025879f32124062b21267b59cb17e661f686 WatchSource:0}: Error finding container ae0c14ab0675025b6798aefc2e90025879f32124062b21267b59cb17e661f686: Status 404 returned error can't find the container with id ae0c14ab0675025b6798aefc2e90025879f32124062b21267b59cb17e661f686
Jun 02 16:02:32.545989 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:32.545952    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg" event={"ID":"345ef462-6ddc-498f-a6ff-0c33412fd7a3","Type":"ContainerStarted","Data":"d8a9846a41690a4d4020a24c7a94956e689a30b993bb613d479080a35d5b47e1"}
Jun 02 16:02:32.545989 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:32.545987    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg" event={"ID":"345ef462-6ddc-498f-a6ff-0c33412fd7a3","Type":"ContainerStarted","Data":"ae0c14ab0675025b6798aefc2e90025879f32124062b21267b59cb17e661f686"}
Jun 02 16:02:32.566845 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:32.566793    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-4l4xg" podStartSLOduration=1.313163477 podStartE2EDuration="1.566779009s" podCreationTimestamp="2026-06-02 16:02:31 +0000 UTC" firstStartedPulling="2026-06-02 16:02:31.746115937 +0000 UTC m=+948.621642075" lastFinishedPulling="2026-06-02 16:02:31.999731464 +0000 UTC m=+948.875257607" observedRunningTime="2026-06-02 16:02:32.564280093 +0000 UTC m=+949.439806255" watchObservedRunningTime="2026-06-02 16:02:32.566779009 +0000 UTC m=+949.442305148"
Jun 02 16:02:52.821256 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.821167    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:02:52.825899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.825874    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:52.829163 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.829140    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jun 02 16:02:52.829293 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.829141    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:02:52.830195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.830171    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:02:52.830311 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.830207    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:02:52.830311 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.830171    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jun 02 16:02:52.839583 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.839560    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:02:52.966743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.966708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kbh\" (UniqueName: \"kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:52.966743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.966755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:52.966958 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.966828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:52.966958 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:52.966945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068001 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.067963    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068173 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.068012    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kbh\" (UniqueName: \"kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068173 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.068041    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068173 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.068065    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.068557    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.068803 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.068783    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.070521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.070504    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.080193 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.080138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kbh\" (UniqueName: \"kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh\") pod \"isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.137155 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.137118    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:02:53.278323 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.278289    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:02:53.283714 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:02:53.283674    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12088c4_d9f5_45d1_8198_391f968f531e.slice/crio-ec5c1d636b2d4079fe911800521bf1ef4c9869e9022a821e5eb04ab2ab5b1907 WatchSource:0}: Error finding container ec5c1d636b2d4079fe911800521bf1ef4c9869e9022a821e5eb04ab2ab5b1907: Status 404 returned error can't find the container with id ec5c1d636b2d4079fe911800521bf1ef4c9869e9022a821e5eb04ab2ab5b1907
Jun 02 16:02:53.615091 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:53.615055    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerStarted","Data":"ec5c1d636b2d4079fe911800521bf1ef4c9869e9022a821e5eb04ab2ab5b1907"}
Jun 02 16:02:57.631756 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:02:57.631718    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerStarted","Data":"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"}
Jun 02 16:03:00.643607 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:00.643574    2571 generic.go:358] "Generic (PLEG): container finished" podID="d12088c4-d9f5-45d1-8198-391f968f531e" containerID="b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17" exitCode=0
Jun 02 16:03:00.643987 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:00.643653    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerDied","Data":"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"}
Jun 02 16:03:15.712896 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:15.712864    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerStarted","Data":"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"}
Jun 02 16:03:18.724898 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:18.724858    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerStarted","Data":"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"}
Jun 02 16:03:21.738953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:21.738913    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerStarted","Data":"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"}
Jun 02 16:03:21.739385 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:21.739169    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:03:21.739385 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:21.739304    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:03:21.740525 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:21.740501    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:21.759722 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:21.759670    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podStartSLOduration=1.6181080140000001 podStartE2EDuration="29.759655257s" podCreationTimestamp="2026-06-02 16:02:52 +0000 UTC" firstStartedPulling="2026-06-02 16:02:53.285522116 +0000 UTC m=+970.161048255" lastFinishedPulling="2026-06-02 16:03:21.427069354 +0000 UTC m=+998.302595498" observedRunningTime="2026-06-02 16:03:21.759031843 +0000 UTC m=+998.634558005" watchObservedRunningTime="2026-06-02 16:03:21.759655257 +0000 UTC m=+998.635181418"
Jun 02 16:03:22.743708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:22.743666    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:03:22.743708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:22.743689    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:22.744701 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:22.744670    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:03:22.747582 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:22.747564    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:03:23.747381 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:23.747326    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:23.747865 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:23.747723    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:03:24.751383 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:24.751320    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:24.751797 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:24.751695    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:03:34.751549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:34.751495    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:34.751982 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:34.751947    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:03:44.751844 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:44.751789    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:44.752259 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:44.752199    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:03:54.751264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:54.751215    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:03:54.751771 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:03:54.751745    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:04.751578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:04.751525    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:04:04.752123 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:04.751852    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:14.751697 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:14.751648    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:04:14.752156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:14.752127    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:24.751609 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:24.751570    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:04:24.752156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:24.751759    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:04:37.784219 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.784180    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:04:37.784859 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.784650    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" containerID="cri-o://364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467" gracePeriod=30
Jun 02 16:04:37.784981 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.784703    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" containerID="cri-o://887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400" gracePeriod=30
Jun 02 16:04:37.785042 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.784785    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" containerID="cri-o://7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5" gracePeriod=30
Jun 02 16:04:37.905432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.905396    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:04:37.908339 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.908316    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:37.910436 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.910414    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jun 02 16:04:37.910558 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.910417    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jun 02 16:04:37.921609 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.921583    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:04:37.970308 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.970272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:37.970522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.970332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:37.970522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.970435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdt7r\" (UniqueName: \"kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:37.970522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:37.970457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.010893 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.010858    2571 generic.go:358] "Generic (PLEG): container finished" podID="d12088c4-d9f5-45d1-8198-391f968f531e" containerID="7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5" exitCode=2
Jun 02 16:04:38.011054 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.010936    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerDied","Data":"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"}
Jun 02 16:04:38.071233 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071137    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.071445 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071308    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.071445 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071413    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qdt7r\" (UniqueName: \"kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.071577 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071447    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.071740 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071719    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.072007 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.071985    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.073921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.073898    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.081488 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.081465    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdt7r\" (UniqueName: \"kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r\") pod \"isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.219121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.219083    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:38.351969 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.351761    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:04:38.354796 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:04:38.354769    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd36088e_78fc_4538_be16_cf371d99c322.slice/crio-02a2f2ee3d0487cee0bd7d082646069805dbca38ef8838a1799cb9cd118e3ce3 WatchSource:0}: Error finding container 02a2f2ee3d0487cee0bd7d082646069805dbca38ef8838a1799cb9cd118e3ce3: Status 404 returned error can't find the container with id 02a2f2ee3d0487cee0bd7d082646069805dbca38ef8838a1799cb9cd118e3ce3
Jun 02 16:04:38.356578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:38.356559    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:04:39.016196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:39.016155    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerStarted","Data":"9977049c165d96a201b93ac2eca420ed52a507d0a1a8bf6075af781b5ce44aff"}
Jun 02 16:04:39.016196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:39.016202    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerStarted","Data":"02a2f2ee3d0487cee0bd7d082646069805dbca38ef8838a1799cb9cd118e3ce3"}
Jun 02 16:04:42.744144 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:42.744103    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:04:43.032229 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:43.032145    2571 generic.go:358] "Generic (PLEG): container finished" podID="cd36088e-78fc-4538-be16-cf371d99c322" containerID="9977049c165d96a201b93ac2eca420ed52a507d0a1a8bf6075af781b5ce44aff" exitCode=0
Jun 02 16:04:43.032229 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:43.032217    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerDied","Data":"9977049c165d96a201b93ac2eca420ed52a507d0a1a8bf6075af781b5ce44aff"}
Jun 02 16:04:43.034459 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:43.034438    2571 generic.go:358] "Generic (PLEG): container finished" podID="d12088c4-d9f5-45d1-8198-391f968f531e" containerID="364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467" exitCode=0
Jun 02 16:04:43.034555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:43.034513    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerDied","Data":"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"}
Jun 02 16:04:44.040276 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.040244    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerStarted","Data":"d30d999dd7a52669e77c3777382a0cc0ebff350800a83a994e1cd3c7c691687b"}
Jun 02 16:04:44.040665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.040285    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerStarted","Data":"d31b8910bb492b6e7b495f798117070edf1858e03c766aa3c16771e063442ac3"}
Jun 02 16:04:44.040665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.040301    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerStarted","Data":"b1adec90f4db5b6bc24fa516fd62770c9ac0e02b50334c756780072db9f713bd"}
Jun 02 16:04:44.040665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.040616    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:44.040759 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.040731    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:44.041821 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.041799    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:04:44.067198 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.066081    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podStartSLOduration=7.066059978 podStartE2EDuration="7.066059978s" podCreationTimestamp="2026-06-02 16:04:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:04:44.064943396 +0000 UTC m=+1080.940469556" watchObservedRunningTime="2026-06-02 16:04:44.066059978 +0000 UTC m=+1080.941586140"
Jun 02 16:04:44.752037 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.751991    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:04:44.752354 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:44.752323    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:45.043745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:45.043642    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:45.043745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:45.043661    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:04:45.044829 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:45.044806    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:46.046743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:46.046688    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:04:46.047195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:46.047169    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:47.744047 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:47.744007    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:04:51.050580 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:51.050554    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:04:51.051075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:51.051041    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:04:51.051405 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:51.051381    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:52.744333 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:52.744279    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:04:52.744733 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:52.744489    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:04:54.751852 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:54.751803    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:04:54.752317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:54.752176    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:04:57.743941 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:04:57.743903    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:05:01.051838 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:01.051793    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:05:01.052264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:01.052201    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:02.744016 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:02.743974    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:05:04.751362 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:04.751321    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:05:04.751799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:04.751492    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:05:04.751799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:04.751653    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:04.751799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:04.751772    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:05:07.744296 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:07.744250    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 02 16:05:07.937819 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:07.937793    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:05:08.040880 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.040798    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location\") pod \"d12088c4-d9f5-45d1-8198-391f968f531e\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") "
Jun 02 16:05:08.041048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.040882    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"d12088c4-d9f5-45d1-8198-391f968f531e\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") "
Jun 02 16:05:08.041048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.040915    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kbh\" (UniqueName: \"kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh\") pod \"d12088c4-d9f5-45d1-8198-391f968f531e\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") "
Jun 02 16:05:08.041048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.040938    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls\") pod \"d12088c4-d9f5-45d1-8198-391f968f531e\" (UID: \"d12088c4-d9f5-45d1-8198-391f968f531e\") "
Jun 02 16:05:08.041299 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.041153    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12088c4-d9f5-45d1-8198-391f968f531e" (UID: "d12088c4-d9f5-45d1-8198-391f968f531e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:05:08.041299 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.041234    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "d12088c4-d9f5-45d1-8198-391f968f531e" (UID: "d12088c4-d9f5-45d1-8198-391f968f531e"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:05:08.043145 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.043119    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12088c4-d9f5-45d1-8198-391f968f531e" (UID: "d12088c4-d9f5-45d1-8198-391f968f531e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:05:08.043235 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.043203    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh" (OuterVolumeSpecName: "kube-api-access-b6kbh") pod "d12088c4-d9f5-45d1-8198-391f968f531e" (UID: "d12088c4-d9f5-45d1-8198-391f968f531e"). InnerVolumeSpecName "kube-api-access-b6kbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:05:08.128953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.128919    2571 generic.go:358] "Generic (PLEG): container finished" podID="d12088c4-d9f5-45d1-8198-391f968f531e" containerID="887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400" exitCode=0
Jun 02 16:05:08.129140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.128961    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerDied","Data":"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"}
Jun 02 16:05:08.129140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.129013    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"
Jun 02 16:05:08.129140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.129020    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx" event={"ID":"d12088c4-d9f5-45d1-8198-391f968f531e","Type":"ContainerDied","Data":"ec5c1d636b2d4079fe911800521bf1ef4c9869e9022a821e5eb04ab2ab5b1907"}
Jun 02 16:05:08.129140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.129045    2571 scope.go:117] "RemoveContainer" containerID="887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"
Jun 02 16:05:08.139801 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.139636    2571 scope.go:117] "RemoveContainer" containerID="7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"
Jun 02 16:05:08.142005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.141984    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12088c4-d9f5-45d1-8198-391f968f531e-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:05:08.142099 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.142010    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12088c4-d9f5-45d1-8198-391f968f531e-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:05:08.142099 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.142024    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kbh\" (UniqueName: \"kubernetes.io/projected/d12088c4-d9f5-45d1-8198-391f968f531e-kube-api-access-b6kbh\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:05:08.142099 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.142035    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12088c4-d9f5-45d1-8198-391f968f531e-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:05:08.147425 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.147396    2571 scope.go:117] "RemoveContainer" containerID="364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"
Jun 02 16:05:08.155472 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.155449    2571 scope.go:117] "RemoveContainer" containerID="b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"
Jun 02 16:05:08.156466 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.156325    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:05:08.161108 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.161083    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6f5f55f9d4-bncgx"]
Jun 02 16:05:08.164267 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.164245    2571 scope.go:117] "RemoveContainer" containerID="887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"
Jun 02 16:05:08.164550 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:05:08.164532    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400\": container with ID starting with 887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400 not found: ID does not exist" containerID="887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"
Jun 02 16:05:08.164613 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.164561    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400"} err="failed to get container status \"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400\": rpc error: code = NotFound desc = could not find container \"887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400\": container with ID starting with 887863078a715fd9a506ae2388ce7c51f1c39d6041988cd45f7a05de18711400 not found: ID does not exist"
Jun 02 16:05:08.164613 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.164583    2571 scope.go:117] "RemoveContainer" containerID="7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"
Jun 02 16:05:08.164807 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:05:08.164791    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5\": container with ID starting with 7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5 not found: ID does not exist" containerID="7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"
Jun 02 16:05:08.164846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.164814    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5"} err="failed to get container status \"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5\": rpc error: code = NotFound desc = could not find container \"7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5\": container with ID starting with 7513edd1b74848b08bfebe6934c84f68b03e6df8ef937901af48b38b85dce0d5 not found: ID does not exist"
Jun 02 16:05:08.164846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.164829    2571 scope.go:117] "RemoveContainer" containerID="364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"
Jun 02 16:05:08.165064 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:05:08.165046    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467\": container with ID starting with 364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467 not found: ID does not exist" containerID="364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"
Jun 02 16:05:08.165117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.165070    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467"} err="failed to get container status \"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467\": rpc error: code = NotFound desc = could not find container \"364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467\": container with ID starting with 364471762e38f1e165d499d4ebb075b3b04d0b8ad6bfa573a4668c80fd80f467 not found: ID does not exist"
Jun 02 16:05:08.165117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.165086    2571 scope.go:117] "RemoveContainer" containerID="b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"
Jun 02 16:05:08.165305 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:05:08.165289    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17\": container with ID starting with b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17 not found: ID does not exist" containerID="b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"
Jun 02 16:05:08.165345 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:08.165310    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17"} err="failed to get container status \"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17\": rpc error: code = NotFound desc = could not find container \"b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17\": container with ID starting with b0ef779f72441c8087688cce06c4681ff1e4cec6ee4a568d51db6f2fb236ee17 not found: ID does not exist"
Jun 02 16:05:09.683973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:09.683941    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" path="/var/lib/kubelet/pods/d12088c4-d9f5-45d1-8198-391f968f531e/volumes"
Jun 02 16:05:11.051834 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:11.051797    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:05:11.052257 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:11.052234    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:21.051528 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:21.051482    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:05:21.051945 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:21.051921    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:31.051069 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:31.051024    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:05:31.051576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:31.051432    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:41.051226 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:41.051179    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:05:41.051710 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:41.051605    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:05:51.051588 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:51.051498    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:05:51.052056 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:05:51.051975    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:03.057990 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.057954    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058319    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="storage-initializer"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058330    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="storage-initializer"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058349    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058354    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058360    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058389    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058403    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy"
Jun 02 16:06:03.058416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058408    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy"
Jun 02 16:06:03.058667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058461    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="agent"
Jun 02 16:06:03.058667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058469    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kserve-container"
Jun 02 16:06:03.058667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.058476    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d12088c4-d9f5-45d1-8198-391f968f531e" containerName="kube-rbac-proxy"
Jun 02 16:06:03.061443 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.061423    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.064142 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.064123    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jun 02 16:06:03.064241 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.064138    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jun 02 16:06:03.073104 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.073079    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:06:03.122744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.122708    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:06:03.123233 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.123190    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" containerID="cri-o://d30d999dd7a52669e77c3777382a0cc0ebff350800a83a994e1cd3c7c691687b" gracePeriod=30
Jun 02 16:06:03.123353 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.123212    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" containerID="cri-o://d31b8910bb492b6e7b495f798117070edf1858e03c766aa3c16771e063442ac3" gracePeriod=30
Jun 02 16:06:03.123353 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.123188    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" containerID="cri-o://b1adec90f4db5b6bc24fa516fd62770c9ac0e02b50334c756780072db9f713bd" gracePeriod=30
Jun 02 16:06:03.222337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.222304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.222529 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.222392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.222529 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.222429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7pr\" (UniqueName: \"kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.325307 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.325223    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.325504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.325323    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.325504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.325393    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7pr\" (UniqueName: \"kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.326058 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.326022    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.328248 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.328221    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.328532 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.328510    2571 generic.go:358] "Generic (PLEG): container finished" podID="cd36088e-78fc-4538-be16-cf371d99c322" containerID="d31b8910bb492b6e7b495f798117070edf1858e03c766aa3c16771e063442ac3" exitCode=2
Jun 02 16:06:03.328609 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.328561    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerDied","Data":"d31b8910bb492b6e7b495f798117070edf1858e03c766aa3c16771e063442ac3"}
Jun 02 16:06:03.334734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.334706    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7pr\" (UniqueName: \"kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr\") pod \"message-dumper-predictor-c7d86bcbd-hxrdn\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.373057 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.373025    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:03.508206 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:03.508082    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:06:03.511027 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:06:03.511005    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a25a5e7_25ea_4109_a043_5c844448673e.slice/crio-fa42f61b2a0908ff8c5ab8ce1895f50d6b25909da2a7fd5960c9074ad09c5879 WatchSource:0}: Error finding container fa42f61b2a0908ff8c5ab8ce1895f50d6b25909da2a7fd5960c9074ad09c5879: Status 404 returned error can't find the container with id fa42f61b2a0908ff8c5ab8ce1895f50d6b25909da2a7fd5960c9074ad09c5879
Jun 02 16:06:04.334197 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:04.334160    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerStarted","Data":"fa42f61b2a0908ff8c5ab8ce1895f50d6b25909da2a7fd5960c9074ad09c5879"}
Jun 02 16:06:05.339214 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:05.339175    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerStarted","Data":"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"}
Jun 02 16:06:05.339214 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:05.339219    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerStarted","Data":"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"}
Jun 02 16:06:05.339655 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:05.339305    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:05.357208 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:05.357159    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" podStartSLOduration=1.3551898279999999 podStartE2EDuration="2.357145948s" podCreationTimestamp="2026-06-02 16:06:03 +0000 UTC" firstStartedPulling="2026-06-02 16:06:03.512799966 +0000 UTC m=+1160.388326106" lastFinishedPulling="2026-06-02 16:06:04.514756087 +0000 UTC m=+1161.390282226" observedRunningTime="2026-06-02 16:06:05.355515797 +0000 UTC m=+1162.231041958" watchObservedRunningTime="2026-06-02 16:06:05.357145948 +0000 UTC m=+1162.232672110"
Jun 02 16:06:06.047723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:06.047677    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:06.342565 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:06.342481    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:06.344430 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:06.344408    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:08.351003 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:08.350963    2571 generic.go:358] "Generic (PLEG): container finished" podID="cd36088e-78fc-4538-be16-cf371d99c322" containerID="b1adec90f4db5b6bc24fa516fd62770c9ac0e02b50334c756780072db9f713bd" exitCode=0
Jun 02 16:06:08.351470 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:08.351031    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerDied","Data":"b1adec90f4db5b6bc24fa516fd62770c9ac0e02b50334c756780072db9f713bd"}
Jun 02 16:06:11.047281 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:11.047242    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:11.051921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:11.051886    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:11.052081 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:11.051992    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:06:13.355509 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:13.355479    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:06:16.047142 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:16.047098    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:16.047542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:16.047237    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:21.047136 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:21.047086    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:21.051854 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:21.051820    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:21.052140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:21.052113    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:06:23.133145 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.133111    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:06:23.136138 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.136112    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.138542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.138513    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jun 02 16:06:23.138661 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.138519    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jun 02 16:06:23.145528 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.145507    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:06:23.195352 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.195307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5gp\" (UniqueName: \"kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.195352 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.195410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.195630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.195436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.195630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.195472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296209 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296170    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5gp\" (UniqueName: \"kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296242    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296262    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296299    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296587 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:06:23.296443    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jun 02 16:06:23.296587 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:06:23.296524    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls podName:fa0d45a9-024c-48da-b76b-2681d9bf8c6a nodeName:}" failed. No retries permitted until 2026-06-02 16:06:23.796499669 +0000 UTC m=+1180.672025808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls") pod "isvc-logger-predictor-58f95b76b7-b8t95" (UID: "fa0d45a9-024c-48da-b76b-2681d9bf8c6a") : secret "isvc-logger-predictor-serving-cert" not found
Jun 02 16:06:23.296696 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296681    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.296975 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.296958    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.306717 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.306696    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5gp\" (UniqueName: \"kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.801060 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.801019    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:23.803497 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:23.803473    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") pod \"isvc-logger-predictor-58f95b76b7-b8t95\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:24.047336 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:24.047301    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:24.184143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:24.184040    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:06:24.186517 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:06:24.186470    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0d45a9_024c_48da_b76b_2681d9bf8c6a.slice/crio-46d5b0bdcb5dfc85113cdbcde736451de91cffe860b8afacd168fd498fd94f45 WatchSource:0}: Error finding container 46d5b0bdcb5dfc85113cdbcde736451de91cffe860b8afacd168fd498fd94f45: Status 404 returned error can't find the container with id 46d5b0bdcb5dfc85113cdbcde736451de91cffe860b8afacd168fd498fd94f45
Jun 02 16:06:24.408522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:24.408433    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerStarted","Data":"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"}
Jun 02 16:06:24.408522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:24.408475    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerStarted","Data":"46d5b0bdcb5dfc85113cdbcde736451de91cffe860b8afacd168fd498fd94f45"}
Jun 02 16:06:26.047554 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:26.047508    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:28.424266 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:28.424230    2571 generic.go:358] "Generic (PLEG): container finished" podID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerID="1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2" exitCode=0
Jun 02 16:06:28.424681 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:28.424301    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerDied","Data":"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"}
Jun 02 16:06:29.430506 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.430468    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerStarted","Data":"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"}
Jun 02 16:06:29.430506 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.430510    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerStarted","Data":"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"}
Jun 02 16:06:29.430991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.430520    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerStarted","Data":"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"}
Jun 02 16:06:29.430991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.430836    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:29.431060 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.430989    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:29.432162 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.432133    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:29.454222 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:29.454173    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podStartSLOduration=6.454159594 podStartE2EDuration="6.454159594s" podCreationTimestamp="2026-06-02 16:06:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:06:29.45329945 +0000 UTC m=+1186.328825612" watchObservedRunningTime="2026-06-02 16:06:29.454159594 +0000 UTC m=+1186.329685755"
Jun 02 16:06:30.434665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:30.434632    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:30.435153 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:30.434672    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:30.435637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:30.435602    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:31.047346 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.047297    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 02 16:06:31.051593 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.051566    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:31.051694 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.051680    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:31.052030 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.052011    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:5000: connect: connection refused"
Jun 02 16:06:31.052101 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.052091    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:31.438395 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.438272    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:31.438769 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:31.438717    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:33.446026 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.445936    2571 generic.go:358] "Generic (PLEG): container finished" podID="cd36088e-78fc-4538-be16-cf371d99c322" containerID="d30d999dd7a52669e77c3777382a0cc0ebff350800a83a994e1cd3c7c691687b" exitCode=0
Jun 02 16:06:33.446026 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.445990    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerDied","Data":"d30d999dd7a52669e77c3777382a0cc0ebff350800a83a994e1cd3c7c691687b"}
Jun 02 16:06:33.780440 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.780414    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:33.888956 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.888918    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdt7r\" (UniqueName: \"kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r\") pod \"cd36088e-78fc-4538-be16-cf371d99c322\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") "
Jun 02 16:06:33.889150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.888999    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"cd36088e-78fc-4538-be16-cf371d99c322\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") "
Jun 02 16:06:33.889150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.889022    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls\") pod \"cd36088e-78fc-4538-be16-cf371d99c322\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") "
Jun 02 16:06:33.889150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.889047    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location\") pod \"cd36088e-78fc-4538-be16-cf371d99c322\" (UID: \"cd36088e-78fc-4538-be16-cf371d99c322\") "
Jun 02 16:06:33.889436 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.889406    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "cd36088e-78fc-4538-be16-cf371d99c322" (UID: "cd36088e-78fc-4538-be16-cf371d99c322"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:06:33.889565 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.889427    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd36088e-78fc-4538-be16-cf371d99c322" (UID: "cd36088e-78fc-4538-be16-cf371d99c322"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:06:33.891177 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.891148    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r" (OuterVolumeSpecName: "kube-api-access-qdt7r") pod "cd36088e-78fc-4538-be16-cf371d99c322" (UID: "cd36088e-78fc-4538-be16-cf371d99c322"). InnerVolumeSpecName "kube-api-access-qdt7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:06:33.891252 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.891184    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd36088e-78fc-4538-be16-cf371d99c322" (UID: "cd36088e-78fc-4538-be16-cf371d99c322"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:06:33.989797 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.989760    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd36088e-78fc-4538-be16-cf371d99c322-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:06:33.989797 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.989793    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd36088e-78fc-4538-be16-cf371d99c322-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:06:33.989991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.989806    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd36088e-78fc-4538-be16-cf371d99c322-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:06:33.989991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:33.989817    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdt7r\" (UniqueName: \"kubernetes.io/projected/cd36088e-78fc-4538-be16-cf371d99c322-kube-api-access-qdt7r\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:06:34.451023 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.450990    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6" event={"ID":"cd36088e-78fc-4538-be16-cf371d99c322","Type":"ContainerDied","Data":"02a2f2ee3d0487cee0bd7d082646069805dbca38ef8838a1799cb9cd118e3ce3"}
Jun 02 16:06:34.451023 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.451027    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"
Jun 02 16:06:34.451529 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.451034    2571 scope.go:117] "RemoveContainer" containerID="d30d999dd7a52669e77c3777382a0cc0ebff350800a83a994e1cd3c7c691687b"
Jun 02 16:06:34.459357 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.459336    2571 scope.go:117] "RemoveContainer" containerID="d31b8910bb492b6e7b495f798117070edf1858e03c766aa3c16771e063442ac3"
Jun 02 16:06:34.466954 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.466932    2571 scope.go:117] "RemoveContainer" containerID="b1adec90f4db5b6bc24fa516fd62770c9ac0e02b50334c756780072db9f713bd"
Jun 02 16:06:34.476888 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.476861    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:06:34.481218 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.481193    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c89d6f497-p4hn6"]
Jun 02 16:06:34.481584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:34.481567    2571 scope.go:117] "RemoveContainer" containerID="9977049c165d96a201b93ac2eca420ed52a507d0a1a8bf6075af781b5ce44aff"
Jun 02 16:06:35.684573 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:35.684541    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd36088e-78fc-4538-be16-cf371d99c322" path="/var/lib/kubelet/pods/cd36088e-78fc-4538-be16-cf371d99c322/volumes"
Jun 02 16:06:36.442956 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:36.442930    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:06:36.443569 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:36.443537    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:36.443852 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:36.443825    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:43.660056 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:43.660027    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:06:43.660475 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:43.660032    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:06:46.444386 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:46.444322    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:46.444857 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:46.444826    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:56.443517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:56.443470    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:06:56.443951 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:06:56.443926    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:06.443503 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:06.443456    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:07:06.443934 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:06.443824    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:16.443818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:16.443767    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:07:16.444286 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:16.444203    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:26.444380 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:26.444316    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:07:26.444876 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:26.444711    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:36.444745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:36.444709    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:07:36.445239 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:36.445130    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:07:48.158784 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.158754    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-hxrdn_9a25a5e7-25ea-4109-a043-5c844448673e/kserve-container/0.log"
Jun 02 16:07:48.340571 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.340537    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:07:48.341032 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.340996    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" containerID="cri-o://0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f" gracePeriod=30
Jun 02 16:07:48.341176 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.341083    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" containerID="cri-o://4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234" gracePeriod=30
Jun 02 16:07:48.341176 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.341089    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" containerID="cri-o://094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2" gracePeriod=30
Jun 02 16:07:48.393538 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.393485    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:07:48.394185 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394164    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy"
Jun 02 16:07:48.394185 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394183    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394194    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394200    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394214    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394228    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="storage-initializer"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394234    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="storage-initializer"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394313    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="agent"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394321    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kserve-container"
Jun 02 16:07:48.394337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.394331    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd36088e-78fc-4538-be16-cf371d99c322" containerName="kube-rbac-proxy"
Jun 02 16:07:48.397898 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.397878    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.400557 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.400535    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jun 02 16:07:48.400669 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.400571    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jun 02 16:07:48.412573 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.412517    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:07:48.457601 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.457570    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:07:48.457909 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.457877    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kserve-container" containerID="cri-o://44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946" gracePeriod=30
Jun 02 16:07:48.458046 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.457943    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kube-rbac-proxy" containerID="cri-o://8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666" gracePeriod=30
Jun 02 16:07:48.534756 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.534726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf4w\" (UniqueName: \"kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.534938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.534776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.534938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.534824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.535036 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.534927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.635981 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.635945    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf4w\" (UniqueName: \"kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.636143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.635994    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.636143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.636021    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.636143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.636043    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.636308 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:07:48.636189    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jun 02 16:07:48.636308 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:07:48.636262    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls podName:71b61cff-e1cc-4c2a-9142-862bd51c833a nodeName:}" failed. No retries permitted until 2026-06-02 16:07:49.136240523 +0000 UTC m=+1266.011766665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls") pod "isvc-lightgbm-predictor-bdf964bd-vzn52" (UID: "71b61cff-e1cc-4c2a-9142-862bd51c833a") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jun 02 16:07:48.636500 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.636461    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.636739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.636723    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.647603 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.647571    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf4w\" (UniqueName: \"kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:48.702156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.702134    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:07:48.712472 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.712447    2571 generic.go:358] "Generic (PLEG): container finished" podID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerID="4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234" exitCode=2
Jun 02 16:07:48.712602 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.712524    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerDied","Data":"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"}
Jun 02 16:07:48.713788 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713771    2571 generic.go:358] "Generic (PLEG): container finished" podID="9a25a5e7-25ea-4109-a043-5c844448673e" containerID="8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666" exitCode=2
Jun 02 16:07:48.713788 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713785    2571 generic.go:358] "Generic (PLEG): container finished" podID="9a25a5e7-25ea-4109-a043-5c844448673e" containerID="44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946" exitCode=2
Jun 02 16:07:48.713912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713811    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerDied","Data":"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"}
Jun 02 16:07:48.713912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713830    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerDied","Data":"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"}
Jun 02 16:07:48.713912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713837    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"
Jun 02 16:07:48.713912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713849    2571 scope.go:117] "RemoveContainer" containerID="8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"
Jun 02 16:07:48.714050 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.713840    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn" event={"ID":"9a25a5e7-25ea-4109-a043-5c844448673e","Type":"ContainerDied","Data":"fa42f61b2a0908ff8c5ab8ce1895f50d6b25909da2a7fd5960c9074ad09c5879"}
Jun 02 16:07:48.722892 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.722873    2571 scope.go:117] "RemoveContainer" containerID="44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"
Jun 02 16:07:48.735275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.735237    2571 scope.go:117] "RemoveContainer" containerID="8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"
Jun 02 16:07:48.735609 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:07:48.735590    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666\": container with ID starting with 8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666 not found: ID does not exist" containerID="8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"
Jun 02 16:07:48.735691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.735630    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"} err="failed to get container status \"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666\": rpc error: code = NotFound desc = could not find container \"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666\": container with ID starting with 8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666 not found: ID does not exist"
Jun 02 16:07:48.735691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.735649    2571 scope.go:117] "RemoveContainer" containerID="44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"
Jun 02 16:07:48.735923 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:07:48.735909    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946\": container with ID starting with 44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946 not found: ID does not exist" containerID="44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"
Jun 02 16:07:48.735971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.735925    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"} err="failed to get container status \"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946\": rpc error: code = NotFound desc = could not find container \"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946\": container with ID starting with 44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946 not found: ID does not exist"
Jun 02 16:07:48.735971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.735937    2571 scope.go:117] "RemoveContainer" containerID="8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"
Jun 02 16:07:48.736196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.736169    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666"} err="failed to get container status \"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666\": rpc error: code = NotFound desc = could not find container \"8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666\": container with ID starting with 8848dcea76be3e9c74ff3d924df8015a43f454fd9a664dc1f503951bd1ff8666 not found: ID does not exist"
Jun 02 16:07:48.736240 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.736199    2571 scope.go:117] "RemoveContainer" containerID="44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"
Jun 02 16:07:48.736504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.736477    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946"} err="failed to get container status \"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946\": rpc error: code = NotFound desc = could not find container \"44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946\": container with ID starting with 44717b4f9189b4e23d0ce5f210730a524e4a8773df299b9e83e76b24ddcf9946 not found: ID does not exist"
Jun 02 16:07:48.837564 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.837522    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config\") pod \"9a25a5e7-25ea-4109-a043-5c844448673e\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") "
Jun 02 16:07:48.837780 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.837640    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl7pr\" (UniqueName: \"kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr\") pod \"9a25a5e7-25ea-4109-a043-5c844448673e\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") "
Jun 02 16:07:48.837780 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.837689    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls\") pod \"9a25a5e7-25ea-4109-a043-5c844448673e\" (UID: \"9a25a5e7-25ea-4109-a043-5c844448673e\") "
Jun 02 16:07:48.843536 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.838973    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "9a25a5e7-25ea-4109-a043-5c844448673e" (UID: "9a25a5e7-25ea-4109-a043-5c844448673e"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:07:48.843536 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.840637    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a25a5e7-25ea-4109-a043-5c844448673e" (UID: "9a25a5e7-25ea-4109-a043-5c844448673e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:07:48.843536 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.841024    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr" (OuterVolumeSpecName: "kube-api-access-fl7pr") pod "9a25a5e7-25ea-4109-a043-5c844448673e" (UID: "9a25a5e7-25ea-4109-a043-5c844448673e"). InnerVolumeSpecName "kube-api-access-fl7pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:07:48.938679 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.938642    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7pr\" (UniqueName: \"kubernetes.io/projected/9a25a5e7-25ea-4109-a043-5c844448673e-kube-api-access-fl7pr\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:07:48.938679 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.938676    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a25a5e7-25ea-4109-a043-5c844448673e-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:07:48.938883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:48.938690    2571 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a25a5e7-25ea-4109-a043-5c844448673e-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:07:49.037982 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.037885    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:07:49.039871 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.039844    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hxrdn"]
Jun 02 16:07:49.139799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.139768    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:49.142138 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.142110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-vzn52\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:49.309315 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.309217    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:07:49.442196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.442052    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:07:49.445038 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:07:49.445004    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b61cff_e1cc_4c2a_9142_862bd51c833a.slice/crio-b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d WatchSource:0}: Error finding container b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d: Status 404 returned error can't find the container with id b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d
Jun 02 16:07:49.684667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.684588    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" path="/var/lib/kubelet/pods/9a25a5e7-25ea-4109-a043-5c844448673e/volumes"
Jun 02 16:07:49.719465 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.719429    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerStarted","Data":"ce0eb21ad688f3f1aaf62c5a6298039c27c6220a053c6fd10f098fed99417ad5"}
Jun 02 16:07:49.719465 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:49.719466    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerStarted","Data":"b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d"}
Jun 02 16:07:51.439256 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:51.439212    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:07:53.735715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:53.735679    2571 generic.go:358] "Generic (PLEG): container finished" podID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerID="0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f" exitCode=0
Jun 02 16:07:53.736136 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:53.735754    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerDied","Data":"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"}
Jun 02 16:07:53.736917 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:53.736897    2571 generic.go:358] "Generic (PLEG): container finished" podID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerID="ce0eb21ad688f3f1aaf62c5a6298039c27c6220a053c6fd10f098fed99417ad5" exitCode=0
Jun 02 16:07:53.737028 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:53.736960    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerDied","Data":"ce0eb21ad688f3f1aaf62c5a6298039c27c6220a053c6fd10f098fed99417ad5"}
Jun 02 16:07:56.439164 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:56.439122    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:07:56.444001 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:56.443960    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:56.445271 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:07:56.445240    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:08:00.773246 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.773199    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerStarted","Data":"f8c7cb20d26d98bed427a54ee3a7efbf250871ffb4bbfef146e77a11d90f613d"}
Jun 02 16:08:00.773246 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.773253    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerStarted","Data":"42cb44d53fa335ef2f13159db00782b90e33a16f189b446a771ce410900daf04"}
Jun 02 16:08:00.773758 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.773612    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:08:00.773811 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.773753    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:08:00.775125 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.775100    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:00.801004 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:00.800880    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podStartSLOduration=5.943235866 podStartE2EDuration="12.800866023s" podCreationTimestamp="2026-06-02 16:07:48 +0000 UTC" firstStartedPulling="2026-06-02 16:07:53.738205417 +0000 UTC m=+1270.613731556" lastFinishedPulling="2026-06-02 16:08:00.595835573 +0000 UTC m=+1277.471361713" observedRunningTime="2026-06-02 16:08:00.79966848 +0000 UTC m=+1277.675194674" watchObservedRunningTime="2026-06-02 16:08:00.800866023 +0000 UTC m=+1277.676392184"
Jun 02 16:08:01.438621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:01.438573    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:08:01.438830 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:01.438787    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:08:01.776936 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:01.776893    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:06.439529 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:06.439483    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:08:06.444202 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:06.444167    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:06.445444 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:06.445416    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:08:06.780727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:06.780699    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:08:06.781379 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:06.781338    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:11.439466 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:11.439416    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:08:16.439048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.439001    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jun 02 16:08:16.444888 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.444853    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:16.445017 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.445001    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:08:16.445191 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.445173    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 02 16:08:16.445262 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.445250    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:08:16.782051 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:16.782007    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:18.493420 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.493394    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:08:18.596484 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596349    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location\") pod \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") "
Jun 02 16:08:18.596484 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596450    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") pod \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") "
Jun 02 16:08:18.596484 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596477    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s5gp\" (UniqueName: \"kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp\") pod \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") "
Jun 02 16:08:18.596803 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596676    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\" (UID: \"fa0d45a9-024c-48da-b76b-2681d9bf8c6a\") "
Jun 02 16:08:18.596803 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596750    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa0d45a9-024c-48da-b76b-2681d9bf8c6a" (UID: "fa0d45a9-024c-48da-b76b-2681d9bf8c6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:08:18.597009 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.596991    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:08:18.597055 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.597013    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "fa0d45a9-024c-48da-b76b-2681d9bf8c6a" (UID: "fa0d45a9-024c-48da-b76b-2681d9bf8c6a"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:08:18.598519 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.598496    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa0d45a9-024c-48da-b76b-2681d9bf8c6a" (UID: "fa0d45a9-024c-48da-b76b-2681d9bf8c6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:08:18.598618 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.598533    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp" (OuterVolumeSpecName: "kube-api-access-7s5gp") pod "fa0d45a9-024c-48da-b76b-2681d9bf8c6a" (UID: "fa0d45a9-024c-48da-b76b-2681d9bf8c6a"). InnerVolumeSpecName "kube-api-access-7s5gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:08:18.698253 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.698211    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:08:18.698253 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.698245    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5gp\" (UniqueName: \"kubernetes.io/projected/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-kube-api-access-7s5gp\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:08:18.698253 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.698255    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d45a9-024c-48da-b76b-2681d9bf8c6a-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:08:18.837876 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.837840    2571 generic.go:358] "Generic (PLEG): container finished" podID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerID="094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2" exitCode=0
Jun 02 16:08:18.838041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.837913    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerDied","Data":"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"}
Jun 02 16:08:18.838041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.837937    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"
Jun 02 16:08:18.838041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.837952    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95" event={"ID":"fa0d45a9-024c-48da-b76b-2681d9bf8c6a","Type":"ContainerDied","Data":"46d5b0bdcb5dfc85113cdbcde736451de91cffe860b8afacd168fd498fd94f45"}
Jun 02 16:08:18.838041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.837968    2571 scope.go:117] "RemoveContainer" containerID="094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"
Jun 02 16:08:18.847243 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.847212    2571 scope.go:117] "RemoveContainer" containerID="4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"
Jun 02 16:08:18.857007 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.856978    2571 scope.go:117] "RemoveContainer" containerID="0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"
Jun 02 16:08:18.861971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.861945    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:08:18.866009 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.865533    2571 scope.go:117] "RemoveContainer" containerID="1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"
Jun 02 16:08:18.867673 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.867653    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58f95b76b7-b8t95"]
Jun 02 16:08:18.872852 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.872834    2571 scope.go:117] "RemoveContainer" containerID="094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"
Jun 02 16:08:18.873138 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:08:18.873119    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2\": container with ID starting with 094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2 not found: ID does not exist" containerID="094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"
Jun 02 16:08:18.873189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873146    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2"} err="failed to get container status \"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2\": rpc error: code = NotFound desc = could not find container \"094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2\": container with ID starting with 094c12e028341533dc5aa97f2e5d9e996fd944ce7d5243f95bb463a6c3db76d2 not found: ID does not exist"
Jun 02 16:08:18.873189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873166    2571 scope.go:117] "RemoveContainer" containerID="4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"
Jun 02 16:08:18.873405 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:08:18.873389    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234\": container with ID starting with 4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234 not found: ID does not exist" containerID="4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"
Jun 02 16:08:18.873460 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873409    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234"} err="failed to get container status \"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234\": rpc error: code = NotFound desc = could not find container \"4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234\": container with ID starting with 4eea162a96368a36f022b79b9c4927d13152f33870cac292cd6b7a8967e57234 not found: ID does not exist"
Jun 02 16:08:18.873460 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873422    2571 scope.go:117] "RemoveContainer" containerID="0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"
Jun 02 16:08:18.873648 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:08:18.873628    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f\": container with ID starting with 0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f not found: ID does not exist" containerID="0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"
Jun 02 16:08:18.873696 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873653    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f"} err="failed to get container status \"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f\": rpc error: code = NotFound desc = could not find container \"0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f\": container with ID starting with 0f91a81ec44347966a2509b4267f6cb826796a84bc25c72c4ccca0339982a86f not found: ID does not exist"
Jun 02 16:08:18.873696 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873665    2571 scope.go:117] "RemoveContainer" containerID="1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"
Jun 02 16:08:18.873847 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:08:18.873819    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2\": container with ID starting with 1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2 not found: ID does not exist" containerID="1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"
Jun 02 16:08:18.873947 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:18.873847    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2"} err="failed to get container status \"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2\": rpc error: code = NotFound desc = could not find container \"1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2\": container with ID starting with 1597c0264955127ac10bd253e40844cdd0a977c7114995e287c7a830ff7c2cd2 not found: ID does not exist"
Jun 02 16:08:19.683914 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:19.683880    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" path="/var/lib/kubelet/pods/fa0d45a9-024c-48da-b76b-2681d9bf8c6a/volumes"
Jun 02 16:08:26.781619 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:26.781572    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:36.781356 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:36.781311    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:46.781856 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:46.781816    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:08:56.781781 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:08:56.781687    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:09:06.781605 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:06.781562    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:09:10.680421 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:10.680347    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:09:20.680547 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:20.680518    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:09:28.554288 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.554251    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:09:28.555247 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.555184    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" containerID="cri-o://42cb44d53fa335ef2f13159db00782b90e33a16f189b446a771ce410900daf04" gracePeriod=30
Jun 02 16:09:28.555359 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.555287    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kube-rbac-proxy" containerID="cri-o://f8c7cb20d26d98bed427a54ee3a7efbf250871ffb4bbfef146e77a11d90f613d" gracePeriod=30
Jun 02 16:09:28.678295 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678259    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:09:28.678775 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678755    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678777    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678789    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678797    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678833    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kube-rbac-proxy"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678844    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kube-rbac-proxy"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678855    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kserve-container"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678865    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kserve-container"
Jun 02 16:09:28.678882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678880    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="storage-initializer"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678890    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="storage-initializer"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678901    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.678911    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.679000    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kserve-container"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.679012    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kserve-container"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.679021    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="agent"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.679032    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="9a25a5e7-25ea-4109-a043-5c844448673e" containerName="kube-rbac-proxy"
Jun 02 16:09:28.679269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.679043    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa0d45a9-024c-48da-b76b-2681d9bf8c6a" containerName="kube-rbac-proxy"
Jun 02 16:09:28.681526 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.681506    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.684194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.684176    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jun 02 16:09:28.684290 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.684206    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:09:28.694551 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.694525    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:09:28.819766 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.819664    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.819766 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.819731    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.819984 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.819767    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnbb\" (UniqueName: \"kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.819984 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.819812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921247 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921209    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921247 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921252    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-chnbb\" (UniqueName: \"kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921540 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921286    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921540 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921340    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921713    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.921972 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.921949    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.923646 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.923627    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.929139 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.929114    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnbb\" (UniqueName: \"kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:28.991792 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:28.991735    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:29.104594 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:29.104556    2571 generic.go:358] "Generic (PLEG): container finished" podID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerID="f8c7cb20d26d98bed427a54ee3a7efbf250871ffb4bbfef146e77a11d90f613d" exitCode=2
Jun 02 16:09:29.104732 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:29.104630    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerDied","Data":"f8c7cb20d26d98bed427a54ee3a7efbf250871ffb4bbfef146e77a11d90f613d"}
Jun 02 16:09:29.123537 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:29.123355    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:09:29.126308 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:09:29.126280    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fa1e2d_d2d9_438b_9bb8_133755acd1d5.slice/crio-1d8370b452eb2bc47c770fc1a2ecb92553748ed5e586d0cfb50d93227ffcc0e6 WatchSource:0}: Error finding container 1d8370b452eb2bc47c770fc1a2ecb92553748ed5e586d0cfb50d93227ffcc0e6: Status 404 returned error can't find the container with id 1d8370b452eb2bc47c770fc1a2ecb92553748ed5e586d0cfb50d93227ffcc0e6
Jun 02 16:09:30.109414 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:30.109346    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerStarted","Data":"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"}
Jun 02 16:09:30.109414 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:30.109407    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerStarted","Data":"1d8370b452eb2bc47c770fc1a2ecb92553748ed5e586d0cfb50d93227ffcc0e6"}
Jun 02 16:09:30.681233 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:30.681195    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jun 02 16:09:31.777951 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:31.777903    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jun 02 16:09:33.119561 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.119531    2571 generic.go:358] "Generic (PLEG): container finished" podID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerID="f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1" exitCode=0
Jun 02 16:09:33.119964 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.119608    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerDied","Data":"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"}
Jun 02 16:09:33.121467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.121443    2571 generic.go:358] "Generic (PLEG): container finished" podID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerID="42cb44d53fa335ef2f13159db00782b90e33a16f189b446a771ce410900daf04" exitCode=0
Jun 02 16:09:33.121584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.121484    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerDied","Data":"42cb44d53fa335ef2f13159db00782b90e33a16f189b446a771ce410900daf04"}
Jun 02 16:09:33.121584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.121509    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52" event={"ID":"71b61cff-e1cc-4c2a-9142-862bd51c833a","Type":"ContainerDied","Data":"b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d"}
Jun 02 16:09:33.121584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.121523    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b63f2e33d16307defaa655e36c72ac5dcb923e1179585d94842bec0840754d3d"
Jun 02 16:09:33.124972 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.124955    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:09:33.258348 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258322    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") pod \"71b61cff-e1cc-4c2a-9142-862bd51c833a\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") "
Jun 02 16:09:33.258536 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258514    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location\") pod \"71b61cff-e1cc-4c2a-9142-862bd51c833a\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") "
Jun 02 16:09:33.258581 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258545    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"71b61cff-e1cc-4c2a-9142-862bd51c833a\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") "
Jun 02 16:09:33.258619 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258595    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pf4w\" (UniqueName: \"kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w\") pod \"71b61cff-e1cc-4c2a-9142-862bd51c833a\" (UID: \"71b61cff-e1cc-4c2a-9142-862bd51c833a\") "
Jun 02 16:09:33.258870 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258833    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71b61cff-e1cc-4c2a-9142-862bd51c833a" (UID: "71b61cff-e1cc-4c2a-9142-862bd51c833a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:09:33.258986 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258907    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b61cff-e1cc-4c2a-9142-862bd51c833a-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:09:33.258986 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.258967    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "71b61cff-e1cc-4c2a-9142-862bd51c833a" (UID: "71b61cff-e1cc-4c2a-9142-862bd51c833a"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:09:33.260282 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.260261    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71b61cff-e1cc-4c2a-9142-862bd51c833a" (UID: "71b61cff-e1cc-4c2a-9142-862bd51c833a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:09:33.260529 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.260513    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w" (OuterVolumeSpecName: "kube-api-access-5pf4w") pod "71b61cff-e1cc-4c2a-9142-862bd51c833a" (UID: "71b61cff-e1cc-4c2a-9142-862bd51c833a"). InnerVolumeSpecName "kube-api-access-5pf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:09:33.359768 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.359733    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b61cff-e1cc-4c2a-9142-862bd51c833a-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:09:33.359768 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.359764    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pf4w\" (UniqueName: \"kubernetes.io/projected/71b61cff-e1cc-4c2a-9142-862bd51c833a-kube-api-access-5pf4w\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:09:33.359768 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:33.359774    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b61cff-e1cc-4c2a-9142-862bd51c833a-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:09:34.127757 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.127725    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerStarted","Data":"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"}
Jun 02 16:09:34.128170 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.127765    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerStarted","Data":"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"}
Jun 02 16:09:34.128170 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.127789    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"
Jun 02 16:09:34.128407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.128358    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:34.129916 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.129794    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:09:34.153175 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.153126    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podStartSLOduration=6.153112791 podStartE2EDuration="6.153112791s" podCreationTimestamp="2026-06-02 16:09:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:09:34.152676811 +0000 UTC m=+1371.028202974" watchObservedRunningTime="2026-06-02 16:09:34.153112791 +0000 UTC m=+1371.028638951"
Jun 02 16:09:34.169894 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.169862    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:09:34.172033 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:34.172010    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-vzn52"]
Jun 02 16:09:35.132048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:35.131957    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:35.132048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:35.131969    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:09:35.684553 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:35.684520    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" path="/var/lib/kubelet/pods/71b61cff-e1cc-4c2a-9142-862bd51c833a/volumes"
Jun 02 16:09:36.135736 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:36.135700    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:09:41.141798 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:41.141768    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:09:41.142409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:41.142356    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:09:51.143130 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:09:51.143086    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:01.143138 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:01.143093    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:11.142417 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:11.142361    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:21.143187 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:21.143142    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:31.143288 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:31.143248    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:41.143129 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:41.143086    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:42.680696 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:42.680650    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:10:52.681590 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:52.681550    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:10:59.289230 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.289188    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:10:59.289699 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.289650    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" containerID="cri-o://1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a" gracePeriod=30
Jun 02 16:10:59.289793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.289704    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kube-rbac-proxy" containerID="cri-o://70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906" gracePeriod=30
Jun 02 16:10:59.399657 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.399614    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:10:59.400076 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400059    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container"
Jun 02 16:10:59.400131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400079    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container"
Jun 02 16:10:59.400131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400097    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kube-rbac-proxy"
Jun 02 16:10:59.400131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400103    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kube-rbac-proxy"
Jun 02 16:10:59.400131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400111    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="storage-initializer"
Jun 02 16:10:59.400131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400117    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="storage-initializer"
Jun 02 16:10:59.400335 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400171    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kserve-container"
Jun 02 16:10:59.400335 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.400183    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="71b61cff-e1cc-4c2a-9142-862bd51c833a" containerName="kube-rbac-proxy"
Jun 02 16:10:59.403463 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.403443    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.405747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.405726    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:10:59.405840 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.405744    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jun 02 16:10:59.414857 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.414836    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:10:59.422410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.422324    2571 generic.go:358] "Generic (PLEG): container finished" podID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerID="70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906" exitCode=2
Jun 02 16:10:59.422512 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.422426    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerDied","Data":"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"}
Jun 02 16:10:59.496342 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.496310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.496542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.496350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.496542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.496397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbjs\" (UniqueName: \"kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.496542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.496483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.597741 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.597655    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbjs\" (UniqueName: \"kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.597741 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.597713    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.597978 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.597817    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.597978 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.597849    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.598171 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.598150    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.598319 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.598301    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.600219 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.600201    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.605736 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.605710    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbjs\" (UniqueName: \"kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.713991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.713951    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:10:59.845078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.844997    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:10:59.847785 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:10:59.847720    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9f9204_dbf1_42ea_91ab_2061542c0b30.slice/crio-a55ac566041da49b2357aaf20c2772a585f85a90d3070f20d4570783606bae99 WatchSource:0}: Error finding container a55ac566041da49b2357aaf20c2772a585f85a90d3070f20d4570783606bae99: Status 404 returned error can't find the container with id a55ac566041da49b2357aaf20c2772a585f85a90d3070f20d4570783606bae99
Jun 02 16:10:59.849619 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:10:59.849594    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:11:00.427463 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:00.427426    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerStarted","Data":"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"}
Jun 02 16:11:00.427846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:00.427469    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerStarted","Data":"a55ac566041da49b2357aaf20c2772a585f85a90d3070f20d4570783606bae99"}
Jun 02 16:11:01.137044 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:01.136998    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 02 16:11:02.680705 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:02.680666    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:11:04.334969 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.334945    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:11:04.438549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438510    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls\") pod \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") "
Jun 02 16:11:04.438745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438565    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location\") pod \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") "
Jun 02 16:11:04.438745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438603    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") "
Jun 02 16:11:04.438745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438623    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chnbb\" (UniqueName: \"kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb\") pod \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\" (UID: \"44fa1e2d-d2d9-438b-9bb8-133755acd1d5\") "
Jun 02 16:11:04.439008 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438980    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44fa1e2d-d2d9-438b-9bb8-133755acd1d5" (UID: "44fa1e2d-d2d9-438b-9bb8-133755acd1d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:11:04.439071 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.438982    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "44fa1e2d-d2d9-438b-9bb8-133755acd1d5" (UID: "44fa1e2d-d2d9-438b-9bb8-133755acd1d5"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:11:04.440738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.440714    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44fa1e2d-d2d9-438b-9bb8-133755acd1d5" (UID: "44fa1e2d-d2d9-438b-9bb8-133755acd1d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:11:04.440858 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.440832    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb" (OuterVolumeSpecName: "kube-api-access-chnbb") pod "44fa1e2d-d2d9-438b-9bb8-133755acd1d5" (UID: "44fa1e2d-d2d9-438b-9bb8-133755acd1d5"). InnerVolumeSpecName "kube-api-access-chnbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:11:04.444142 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.444116    2571 generic.go:358] "Generic (PLEG): container finished" podID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerID="1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a" exitCode=0
Jun 02 16:11:04.444237 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.444193    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"
Jun 02 16:11:04.444237 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.444207    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerDied","Data":"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"}
Jun 02 16:11:04.444347 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.444237    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb" event={"ID":"44fa1e2d-d2d9-438b-9bb8-133755acd1d5","Type":"ContainerDied","Data":"1d8370b452eb2bc47c770fc1a2ecb92553748ed5e586d0cfb50d93227ffcc0e6"}
Jun 02 16:11:04.444347 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.444259    2571 scope.go:117] "RemoveContainer" containerID="70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"
Jun 02 16:11:04.445712 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.445690    2571 generic.go:358] "Generic (PLEG): container finished" podID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerID="e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f" exitCode=0
Jun 02 16:11:04.445812 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.445725    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerDied","Data":"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"}
Jun 02 16:11:04.453471 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.453415    2571 scope.go:117] "RemoveContainer" containerID="1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"
Jun 02 16:11:04.461008 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.460986    2571 scope.go:117] "RemoveContainer" containerID="f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"
Jun 02 16:11:04.468150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468132    2571 scope.go:117] "RemoveContainer" containerID="70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"
Jun 02 16:11:04.468417 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:11:04.468389    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906\": container with ID starting with 70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906 not found: ID does not exist" containerID="70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"
Jun 02 16:11:04.468509 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468425    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906"} err="failed to get container status \"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906\": rpc error: code = NotFound desc = could not find container \"70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906\": container with ID starting with 70fb540fd4468019efff635d9cc3b5bcf10f418913c1c9d71fbb7ce6974ba906 not found: ID does not exist"
Jun 02 16:11:04.468509 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468445    2571 scope.go:117] "RemoveContainer" containerID="1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"
Jun 02 16:11:04.468675 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:11:04.468656    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a\": container with ID starting with 1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a not found: ID does not exist" containerID="1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"
Jun 02 16:11:04.468718 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468681    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a"} err="failed to get container status \"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a\": rpc error: code = NotFound desc = could not find container \"1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a\": container with ID starting with 1c77ee61b80f31e973e65f0b6a9341c8b7a438a299b94f129dce475f0d78eb6a not found: ID does not exist"
Jun 02 16:11:04.468718 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468698    2571 scope.go:117] "RemoveContainer" containerID="f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"
Jun 02 16:11:04.468916 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:11:04.468897    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1\": container with ID starting with f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1 not found: ID does not exist" containerID="f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"
Jun 02 16:11:04.468962 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.468922    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1"} err="failed to get container status \"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1\": rpc error: code = NotFound desc = could not find container \"f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1\": container with ID starting with f217c41636b09b78f92a61895091b0fa8c4f6190d0d1b5caaf6212aab3f475d1 not found: ID does not exist"
Jun 02 16:11:04.484454 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.484427    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:11:04.488467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.488443    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4x8hb"]
Jun 02 16:11:04.539833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.539801    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:11:04.539833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.539830    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:11:04.540034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.539846    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:11:04.540034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:04.539862    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnbb\" (UniqueName: \"kubernetes.io/projected/44fa1e2d-d2d9-438b-9bb8-133755acd1d5-kube-api-access-chnbb\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:11:05.689597 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:05.689562    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" path="/var/lib/kubelet/pods/44fa1e2d-d2d9-438b-9bb8-133755acd1d5/volumes"
Jun 02 16:11:43.694761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:43.694726    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:11:43.695324 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:11:43.694826    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:13:38.037831 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:38.037794    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerStarted","Data":"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"}
Jun 02 16:13:38.037831 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:38.037835    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerStarted","Data":"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"}
Jun 02 16:13:38.038359 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:38.037891    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:13:38.038359 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:38.037939    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:13:38.100137 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:38.100070    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" podStartSLOduration=6.225182762 podStartE2EDuration="2m39.100050192s" podCreationTimestamp="2026-06-02 16:10:59 +0000 UTC" firstStartedPulling="2026-06-02 16:11:04.446996686 +0000 UTC m=+1461.322522825" lastFinishedPulling="2026-06-02 16:13:37.321864099 +0000 UTC m=+1614.197390255" observedRunningTime="2026-06-02 16:13:38.098798427 +0000 UTC m=+1614.974324570" watchObservedRunningTime="2026-06-02 16:13:38.100050192 +0000 UTC m=+1614.975576355"
Jun 02 16:13:44.047335 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:13:44.047305    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:14:14.050921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:14.050887    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:14:19.599962 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.599925    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:14:19.600446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.600249    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kserve-container" containerID="cri-o://2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450" gracePeriod=30
Jun 02 16:14:19.600446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.600332    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kube-rbac-proxy" containerID="cri-o://e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a" gracePeriod=30
Jun 02 16:14:19.737402 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737350    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:19.737765 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737750    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container"
Jun 02 16:14:19.737765 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737766    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container"
Jun 02 16:14:19.737848 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737775    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kube-rbac-proxy"
Jun 02 16:14:19.737848 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737781    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kube-rbac-proxy"
Jun 02 16:14:19.737848 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737789    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="storage-initializer"
Jun 02 16:14:19.737848 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737796    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="storage-initializer"
Jun 02 16:14:19.737968 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737871    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kube-rbac-proxy"
Jun 02 16:14:19.737968 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.737879    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="44fa1e2d-d2d9-438b-9bb8-133755acd1d5" containerName="kserve-container"
Jun 02 16:14:19.740057 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.740039    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.744439 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.744408    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 02 16:14:19.754603 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.754576    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jun 02 16:14:19.755477 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.755450    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:19.819383 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.819328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946k4\" (UniqueName: \"kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.819587 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.819488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.819587 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.819562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.819714 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.819630    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.920265 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920153    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.920265 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920231    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-946k4\" (UniqueName: \"kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.920518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920272    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.920518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920326    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.920841 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920809    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.921020 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.920995    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.922974 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.922949    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:19.930711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:19.930684    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946k4\" (UniqueName: \"kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:20.051312 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.051278    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:20.183064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.182934    2571 generic.go:358] "Generic (PLEG): container finished" podID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerID="e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a" exitCode=2
Jun 02 16:14:20.183064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.183015    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerDied","Data":"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"}
Jun 02 16:14:20.185105 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.185084    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:20.187568 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:14:20.187543    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9180a05b_c86f_4831_a105_c7e0782d10a8.slice/crio-fc97e81b73baea0d3660c2f342666dc523075f564436880fc2798f903fc710a5 WatchSource:0}: Error finding container fc97e81b73baea0d3660c2f342666dc523075f564436880fc2798f903fc710a5: Status 404 returned error can't find the container with id fc97e81b73baea0d3660c2f342666dc523075f564436880fc2798f903fc710a5
Jun 02 16:14:20.750624 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.750599    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:14:20.826971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.826871    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") "
Jun 02 16:14:20.826971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.826969    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location\") pod \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") "
Jun 02 16:14:20.827200 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.827002    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpbjs\" (UniqueName: \"kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs\") pod \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") "
Jun 02 16:14:20.827200 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.827031    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls\") pod \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\" (UID: \"fc9f9204-dbf1-42ea-91ab-2061542c0b30\") "
Jun 02 16:14:20.827299 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.827239    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "fc9f9204-dbf1-42ea-91ab-2061542c0b30" (UID: "fc9f9204-dbf1-42ea-91ab-2061542c0b30"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:14:20.827392 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.827325    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9f9204-dbf1-42ea-91ab-2061542c0b30-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:20.827392 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.827348    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc9f9204-dbf1-42ea-91ab-2061542c0b30" (UID: "fc9f9204-dbf1-42ea-91ab-2061542c0b30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:14:20.829114 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.829090    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc9f9204-dbf1-42ea-91ab-2061542c0b30" (UID: "fc9f9204-dbf1-42ea-91ab-2061542c0b30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:14:20.829217 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.829160    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs" (OuterVolumeSpecName: "kube-api-access-cpbjs") pod "fc9f9204-dbf1-42ea-91ab-2061542c0b30" (UID: "fc9f9204-dbf1-42ea-91ab-2061542c0b30"). InnerVolumeSpecName "kube-api-access-cpbjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:14:20.928590 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.928550    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9f9204-dbf1-42ea-91ab-2061542c0b30-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:20.928590 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.928583    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:20.928590 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:20.928594    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbjs\" (UniqueName: \"kubernetes.io/projected/fc9f9204-dbf1-42ea-91ab-2061542c0b30-kube-api-access-cpbjs\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:21.188090 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.187990    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerStarted","Data":"7f124777a8f433b18251cc9f361633bbc2ba6b90eeab01fb026463925d681679"}
Jun 02 16:14:21.188090 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.188031    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerStarted","Data":"fc97e81b73baea0d3660c2f342666dc523075f564436880fc2798f903fc710a5"}
Jun 02 16:14:21.189531 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.189503    2571 generic.go:358] "Generic (PLEG): container finished" podID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerID="2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450" exitCode=0
Jun 02 16:14:21.189665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.189559    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerDied","Data":"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"}
Jun 02 16:14:21.189665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.189583    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx" event={"ID":"fc9f9204-dbf1-42ea-91ab-2061542c0b30","Type":"ContainerDied","Data":"a55ac566041da49b2357aaf20c2772a585f85a90d3070f20d4570783606bae99"}
Jun 02 16:14:21.189665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.189585    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"
Jun 02 16:14:21.189665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.189597    2571 scope.go:117] "RemoveContainer" containerID="e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"
Jun 02 16:14:21.197990 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.197970    2571 scope.go:117] "RemoveContainer" containerID="2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"
Jun 02 16:14:21.205549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.205530    2571 scope.go:117] "RemoveContainer" containerID="e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"
Jun 02 16:14:21.213667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.213510    2571 scope.go:117] "RemoveContainer" containerID="e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"
Jun 02 16:14:21.214010 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:14:21.213873    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a\": container with ID starting with e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a not found: ID does not exist" containerID="e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"
Jun 02 16:14:21.214010 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.213912    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a"} err="failed to get container status \"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a\": rpc error: code = NotFound desc = could not find container \"e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a\": container with ID starting with e9afeacf5396935c9a91e32b3eb97a6c4a88eb6c1c61e6c06d8a0ac93840107a not found: ID does not exist"
Jun 02 16:14:21.214010 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.213935    2571 scope.go:117] "RemoveContainer" containerID="2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"
Jun 02 16:14:21.214252 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:14:21.214228    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450\": container with ID starting with 2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450 not found: ID does not exist" containerID="2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"
Jun 02 16:14:21.214304 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.214264    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450"} err="failed to get container status \"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450\": rpc error: code = NotFound desc = could not find container \"2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450\": container with ID starting with 2bded62347db42a89baf58314c70bb2642a5c90b954dc3e4ee2ee0a485b34450 not found: ID does not exist"
Jun 02 16:14:21.214344 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.214307    2571 scope.go:117] "RemoveContainer" containerID="e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"
Jun 02 16:14:21.214684 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:14:21.214664    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f\": container with ID starting with e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f not found: ID does not exist" containerID="e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"
Jun 02 16:14:21.214684 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.214685    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f"} err="failed to get container status \"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f\": rpc error: code = NotFound desc = could not find container \"e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f\": container with ID starting with e9202e384dbb36c1791f3e681f85adacb35a1109669b41222ae12a081924f86f not found: ID does not exist"
Jun 02 16:14:21.225251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.225215    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:14:21.232162 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.232134    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-4t2wx"]
Jun 02 16:14:21.684087 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:21.684052    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" path="/var/lib/kubelet/pods/fc9f9204-dbf1-42ea-91ab-2061542c0b30/volumes"
Jun 02 16:14:24.201530 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:24.201496    2571 generic.go:358] "Generic (PLEG): container finished" podID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerID="7f124777a8f433b18251cc9f361633bbc2ba6b90eeab01fb026463925d681679" exitCode=0
Jun 02 16:14:24.201907 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:24.201578    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerDied","Data":"7f124777a8f433b18251cc9f361633bbc2ba6b90eeab01fb026463925d681679"}
Jun 02 16:14:25.207453 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.207339    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerStarted","Data":"99a9f211d5bae7432bde7c3b146285609c18942e48fb4bcaa2b90abde131a119"}
Jun 02 16:14:25.207453 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.207413    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerStarted","Data":"f38ed40d74787b0efba7bbd69ae0f5cd38be7549b626e07f0e8f799f1e290cae"}
Jun 02 16:14:25.207938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.207737    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:25.207938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.207878    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:25.209279 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.209257    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 02 16:14:25.235645 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:25.235592    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podStartSLOduration=6.235576096 podStartE2EDuration="6.235576096s" podCreationTimestamp="2026-06-02 16:14:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:14:25.233813949 +0000 UTC m=+1662.109340112" watchObservedRunningTime="2026-06-02 16:14:25.235576096 +0000 UTC m=+1662.111102256"
Jun 02 16:14:26.212823 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:26.212776    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 02 16:14:31.217670 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:31.217639    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:31.218294 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:31.218265    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 02 16:14:41.219093 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:41.219053    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:43.741301 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:43.741268    2571 scope.go:117] "RemoveContainer" containerID="f8c7cb20d26d98bed427a54ee3a7efbf250871ffb4bbfef146e77a11d90f613d"
Jun 02 16:14:43.749912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:43.749894    2571 scope.go:117] "RemoveContainer" containerID="42cb44d53fa335ef2f13159db00782b90e33a16f189b446a771ce410900daf04"
Jun 02 16:14:43.757257 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:43.757237    2571 scope.go:117] "RemoveContainer" containerID="ce0eb21ad688f3f1aaf62c5a6298039c27c6220a053c6fd10f098fed99417ad5"
Jun 02 16:14:49.739968 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.739935    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:49.740428 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.740268    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container" containerID="cri-o://f38ed40d74787b0efba7bbd69ae0f5cd38be7549b626e07f0e8f799f1e290cae" gracePeriod=30
Jun 02 16:14:49.740428 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.740344    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kube-rbac-proxy" containerID="cri-o://99a9f211d5bae7432bde7c3b146285609c18942e48fb4bcaa2b90abde131a119" gracePeriod=30
Jun 02 16:14:49.826901 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.826868    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:14:49.827260 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827248    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="storage-initializer"
Jun 02 16:14:49.827313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827262    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="storage-initializer"
Jun 02 16:14:49.827313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827283    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kube-rbac-proxy"
Jun 02 16:14:49.827313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827290    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kube-rbac-proxy"
Jun 02 16:14:49.827313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827308    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kserve-container"
Jun 02 16:14:49.827313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827314    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kserve-container"
Jun 02 16:14:49.827491 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827388    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kserve-container"
Jun 02 16:14:49.827491 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.827399    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="fc9f9204-dbf1-42ea-91ab-2061542c0b30" containerName="kube-rbac-proxy"
Jun 02 16:14:49.831448 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.831418    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.833923 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.833765    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jun 02 16:14:49.833923 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.833767    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:14:49.839138 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.839111    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:14:49.875608 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.875573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.875761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.875622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.875761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.875647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.875761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.875682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.976750 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.976706    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.976750 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.976755    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.977018 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.976772    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.977018 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.976794    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.977018 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:14:49.976883    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jun 02 16:14:49.977018 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:14:49.976971    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls podName:50317dae-00ec-4fed-bc11-90f789b217b3 nodeName:}" failed. No retries permitted until 2026-06-02 16:14:50.476948932 +0000 UTC m=+1687.352475074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" (UID: "50317dae-00ec-4fed-bc11-90f789b217b3") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jun 02 16:14:49.977213 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.977195    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.977447 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.977429    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:49.985431 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:49.985402    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:50.302684 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.302649    2571 generic.go:358] "Generic (PLEG): container finished" podID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerID="99a9f211d5bae7432bde7c3b146285609c18942e48fb4bcaa2b90abde131a119" exitCode=2
Jun 02 16:14:50.302684 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.302677    2571 generic.go:358] "Generic (PLEG): container finished" podID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerID="f38ed40d74787b0efba7bbd69ae0f5cd38be7549b626e07f0e8f799f1e290cae" exitCode=0
Jun 02 16:14:50.302960 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.302726    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerDied","Data":"99a9f211d5bae7432bde7c3b146285609c18942e48fb4bcaa2b90abde131a119"}
Jun 02 16:14:50.302960 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.302764    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerDied","Data":"f38ed40d74787b0efba7bbd69ae0f5cd38be7549b626e07f0e8f799f1e290cae"}
Jun 02 16:14:50.387606 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.387584    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:50.482446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482324    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"9180a05b-c86f-4831-a105-c7e0782d10a8\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") "
Jun 02 16:14:50.482446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482419    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls\") pod \"9180a05b-c86f-4831-a105-c7e0782d10a8\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") "
Jun 02 16:14:50.482446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482452    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-946k4\" (UniqueName: \"kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4\") pod \"9180a05b-c86f-4831-a105-c7e0782d10a8\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") "
Jun 02 16:14:50.482741 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482487    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location\") pod \"9180a05b-c86f-4831-a105-c7e0782d10a8\" (UID: \"9180a05b-c86f-4831-a105-c7e0782d10a8\") "
Jun 02 16:14:50.482802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482738    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "9180a05b-c86f-4831-a105-c7e0782d10a8" (UID: "9180a05b-c86f-4831-a105-c7e0782d10a8"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:14:50.482802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482749    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:50.482912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482860    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9180a05b-c86f-4831-a105-c7e0782d10a8-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:50.482912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.482883    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9180a05b-c86f-4831-a105-c7e0782d10a8" (UID: "9180a05b-c86f-4831-a105-c7e0782d10a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:14:50.484604 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.484575    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9180a05b-c86f-4831-a105-c7e0782d10a8" (UID: "9180a05b-c86f-4831-a105-c7e0782d10a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:14:50.484604 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.484585    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4" (OuterVolumeSpecName: "kube-api-access-946k4") pod "9180a05b-c86f-4831-a105-c7e0782d10a8" (UID: "9180a05b-c86f-4831-a105-c7e0782d10a8"). InnerVolumeSpecName "kube-api-access-946k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:14:50.484987 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.484967    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:50.583934 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.583896    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9180a05b-c86f-4831-a105-c7e0782d10a8-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:50.583934 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.583928    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946k4\" (UniqueName: \"kubernetes.io/projected/9180a05b-c86f-4831-a105-c7e0782d10a8-kube-api-access-946k4\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:50.583934 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.583937    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9180a05b-c86f-4831-a105-c7e0782d10a8-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:14:50.743908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.743821    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:50.879011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:50.878976    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:14:50.881859 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:14:50.881823    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50317dae_00ec_4fed_bc11_90f789b217b3.slice/crio-54d7a3a80066366098282ce22faa9dd28d49a4d94b2cb24fc0a9152f9587b807 WatchSource:0}: Error finding container 54d7a3a80066366098282ce22faa9dd28d49a4d94b2cb24fc0a9152f9587b807: Status 404 returned error can't find the container with id 54d7a3a80066366098282ce22faa9dd28d49a4d94b2cb24fc0a9152f9587b807
Jun 02 16:14:51.307830 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.307789    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerStarted","Data":"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"}
Jun 02 16:14:51.307830 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.307834    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerStarted","Data":"54d7a3a80066366098282ce22faa9dd28d49a4d94b2cb24fc0a9152f9587b807"}
Jun 02 16:14:51.309616 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.309580    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl" event={"ID":"9180a05b-c86f-4831-a105-c7e0782d10a8","Type":"ContainerDied","Data":"fc97e81b73baea0d3660c2f342666dc523075f564436880fc2798f903fc710a5"}
Jun 02 16:14:51.309616 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.309616    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"
Jun 02 16:14:51.309784 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.309622    2571 scope.go:117] "RemoveContainer" containerID="99a9f211d5bae7432bde7c3b146285609c18942e48fb4bcaa2b90abde131a119"
Jun 02 16:14:51.318783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.318763    2571 scope.go:117] "RemoveContainer" containerID="f38ed40d74787b0efba7bbd69ae0f5cd38be7549b626e07f0e8f799f1e290cae"
Jun 02 16:14:51.327674 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.327650    2571 scope.go:117] "RemoveContainer" containerID="7f124777a8f433b18251cc9f361633bbc2ba6b90eeab01fb026463925d681679"
Jun 02 16:14:51.338960 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.338931    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:51.344387 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.344341    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vvfcl"]
Jun 02 16:14:51.684389 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:51.684287    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" path="/var/lib/kubelet/pods/9180a05b-c86f-4831-a105-c7e0782d10a8/volumes"
Jun 02 16:14:55.327271 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:55.327187    2571 generic.go:358] "Generic (PLEG): container finished" podID="50317dae-00ec-4fed-bc11-90f789b217b3" containerID="f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94" exitCode=0
Jun 02 16:14:55.327271 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:55.327255    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerDied","Data":"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"}
Jun 02 16:14:56.332183 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:56.332146    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerStarted","Data":"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"}
Jun 02 16:14:56.332183 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:56.332189    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerStarted","Data":"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"}
Jun 02 16:14:56.332621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:56.332603    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:14:56.354083 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:56.354020    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" podStartSLOduration=7.353995342 podStartE2EDuration="7.353995342s" podCreationTimestamp="2026-06-02 16:14:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:14:56.351915963 +0000 UTC m=+1693.227442124" watchObservedRunningTime="2026-06-02 16:14:56.353995342 +0000 UTC m=+1693.229521504"
Jun 02 16:14:57.336063 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:14:57.336033    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:15:03.345151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:03.345123    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:15:33.348786 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:33.348756    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:15:39.908562 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:39.908523    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:15:39.909067 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:39.908861    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kserve-container" containerID="cri-o://4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72" gracePeriod=30
Jun 02 16:15:39.909067 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:39.908899    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kube-rbac-proxy" containerID="cri-o://3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5" gracePeriod=30
Jun 02 16:15:40.086750 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.086716    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:15:40.087133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087118    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kube-rbac-proxy"
Jun 02 16:15:40.087194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087138    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kube-rbac-proxy"
Jun 02 16:15:40.087194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087150    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container"
Jun 02 16:15:40.087194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087156    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container"
Jun 02 16:15:40.087194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087170    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="storage-initializer"
Jun 02 16:15:40.087194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087177    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="storage-initializer"
Jun 02 16:15:40.087362 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087240    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kserve-container"
Jun 02 16:15:40.087362 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.087247    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="9180a05b-c86f-4831-a105-c7e0782d10a8" containerName="kube-rbac-proxy"
Jun 02 16:15:40.089709 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.089692    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.092005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.091977    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jun 02 16:15:40.092121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.092062    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jun 02 16:15:40.100578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.100545    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:15:40.212777 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.212750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.212952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.212795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.212952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.212817    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv42d\" (UniqueName: \"kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.213038 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.213000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.314361 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.314324    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.314580 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.314400    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.314580 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.314429    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vv42d\" (UniqueName: \"kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.314580 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.314532    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.314905 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.314880    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.315204 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.315180    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.316812 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.316791    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.322770 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.322744    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv42d\" (UniqueName: \"kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d\") pod \"isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.400791 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.400754    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:40.488451 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.488422    2571 generic.go:358] "Generic (PLEG): container finished" podID="50317dae-00ec-4fed-bc11-90f789b217b3" containerID="3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5" exitCode=2
Jun 02 16:15:40.488609 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.488493    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerDied","Data":"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"}
Jun 02 16:15:40.554631 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:40.554598    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:15:40.559037 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:15:40.559008    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bdae6f_4133_462b_a1d9_27c141c600b6.slice/crio-070fd320e4ad73dbea3110c2db1d5bf44e956a91dbdd4679be96e9f9ef75a49f WatchSource:0}: Error finding container 070fd320e4ad73dbea3110c2db1d5bf44e956a91dbdd4679be96e9f9ef75a49f: Status 404 returned error can't find the container with id 070fd320e4ad73dbea3110c2db1d5bf44e956a91dbdd4679be96e9f9ef75a49f
Jun 02 16:15:41.247558 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.247523    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:15:41.323300 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323201    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location\") pod \"50317dae-00ec-4fed-bc11-90f789b217b3\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") "
Jun 02 16:15:41.323505 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323308    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"50317dae-00ec-4fed-bc11-90f789b217b3\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") "
Jun 02 16:15:41.323505 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323349    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25\") pod \"50317dae-00ec-4fed-bc11-90f789b217b3\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") "
Jun 02 16:15:41.323505 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323415    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") pod \"50317dae-00ec-4fed-bc11-90f789b217b3\" (UID: \"50317dae-00ec-4fed-bc11-90f789b217b3\") "
Jun 02 16:15:41.323683 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323646    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50317dae-00ec-4fed-bc11-90f789b217b3" (UID: "50317dae-00ec-4fed-bc11-90f789b217b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:15:41.323730 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.323688    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "50317dae-00ec-4fed-bc11-90f789b217b3" (UID: "50317dae-00ec-4fed-bc11-90f789b217b3"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:15:41.325452 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.325398    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25" (OuterVolumeSpecName: "kube-api-access-nxq25") pod "50317dae-00ec-4fed-bc11-90f789b217b3" (UID: "50317dae-00ec-4fed-bc11-90f789b217b3"). InnerVolumeSpecName "kube-api-access-nxq25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:15:41.325562 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.325508    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50317dae-00ec-4fed-bc11-90f789b217b3" (UID: "50317dae-00ec-4fed-bc11-90f789b217b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:15:41.424883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.424831    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50317dae-00ec-4fed-bc11-90f789b217b3-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:15:41.424883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.424874    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/50317dae-00ec-4fed-bc11-90f789b217b3-kube-api-access-nxq25\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:15:41.424883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.424886    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50317dae-00ec-4fed-bc11-90f789b217b3-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:15:41.424883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.424897    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50317dae-00ec-4fed-bc11-90f789b217b3-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:15:41.492849 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.492813    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerStarted","Data":"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"}
Jun 02 16:15:41.493032 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.492854    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerStarted","Data":"070fd320e4ad73dbea3110c2db1d5bf44e956a91dbdd4679be96e9f9ef75a49f"}
Jun 02 16:15:41.494521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.494495    2571 generic.go:358] "Generic (PLEG): container finished" podID="50317dae-00ec-4fed-bc11-90f789b217b3" containerID="4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72" exitCode=0
Jun 02 16:15:41.494629 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.494539    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerDied","Data":"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"}
Jun 02 16:15:41.494629 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.494565    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td" event={"ID":"50317dae-00ec-4fed-bc11-90f789b217b3","Type":"ContainerDied","Data":"54d7a3a80066366098282ce22faa9dd28d49a4d94b2cb24fc0a9152f9587b807"}
Jun 02 16:15:41.494629 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.494577    2571 scope.go:117] "RemoveContainer" containerID="3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"
Jun 02 16:15:41.494734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.494564    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"
Jun 02 16:15:41.503001 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.502983    2571 scope.go:117] "RemoveContainer" containerID="4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"
Jun 02 16:15:41.510224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.510206    2571 scope.go:117] "RemoveContainer" containerID="f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"
Jun 02 16:15:41.517872 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.517833    2571 scope.go:117] "RemoveContainer" containerID="3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"
Jun 02 16:15:41.518144 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:15:41.518115    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5\": container with ID starting with 3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5 not found: ID does not exist" containerID="3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"
Jun 02 16:15:41.518241 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.518154    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5"} err="failed to get container status \"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5\": rpc error: code = NotFound desc = could not find container \"3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5\": container with ID starting with 3bc4bbc66d145e7c22214aa1b55833cc5f568b66a9e2741396b6f5c8c3a3b5a5 not found: ID does not exist"
Jun 02 16:15:41.518241 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.518180    2571 scope.go:117] "RemoveContainer" containerID="4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"
Jun 02 16:15:41.518551 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:15:41.518530    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72\": container with ID starting with 4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72 not found: ID does not exist" containerID="4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"
Jun 02 16:15:41.518630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.518560    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72"} err="failed to get container status \"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72\": rpc error: code = NotFound desc = could not find container \"4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72\": container with ID starting with 4b3bcf09d6be5123d4937fd541e8dd15021824f64d764ab10110dd6cc8df1d72 not found: ID does not exist"
Jun 02 16:15:41.518630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.518577    2571 scope.go:117] "RemoveContainer" containerID="f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"
Jun 02 16:15:41.518880 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:15:41.518857    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94\": container with ID starting with f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94 not found: ID does not exist" containerID="f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"
Jun 02 16:15:41.518951 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.518890    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94"} err="failed to get container status \"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94\": rpc error: code = NotFound desc = could not find container \"f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94\": container with ID starting with f5e38155622b0aae9b7901bcbc5c8f9bc234b2439c073af4f39dee36258a4f94 not found: ID does not exist"
Jun 02 16:15:41.537341 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.537313    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:15:41.549445 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.549416    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-579td"]
Jun 02 16:15:41.684787 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:41.684694    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" path="/var/lib/kubelet/pods/50317dae-00ec-4fed-bc11-90f789b217b3/volumes"
Jun 02 16:15:44.507877 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:44.507840    2571 generic.go:358] "Generic (PLEG): container finished" podID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerID="746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa" exitCode=0
Jun 02 16:15:44.508243 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:44.507911    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerDied","Data":"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"}
Jun 02 16:15:45.514835 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:45.514800    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerStarted","Data":"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"}
Jun 02 16:15:47.525651 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:47.525615    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerStarted","Data":"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"}
Jun 02 16:15:47.525651 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:47.525654    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerStarted","Data":"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"}
Jun 02 16:15:47.526110 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:47.525925    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:47.570933 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:47.570881    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podStartSLOduration=5.118052571 podStartE2EDuration="7.570866198s" podCreationTimestamp="2026-06-02 16:15:40 +0000 UTC" firstStartedPulling="2026-06-02 16:15:44.572633009 +0000 UTC m=+1741.448159148" lastFinishedPulling="2026-06-02 16:15:47.025446635 +0000 UTC m=+1743.900972775" observedRunningTime="2026-06-02 16:15:47.567666265 +0000 UTC m=+1744.443192416" watchObservedRunningTime="2026-06-02 16:15:47.570866198 +0000 UTC m=+1744.446392358"
Jun 02 16:15:48.529117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:48.529080    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:48.529117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:48.529120    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:15:54.538755 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:15:54.538723    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:16:14.540333 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:16:14.540286    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 02 16:16:24.540921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:16:24.540888    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:16:43.719953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:16:43.719925    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:16:43.720749 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:16:43.720727    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:16:54.542020 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:16:54.541985    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:17:00.055928 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.055891    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:17:00.056678 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.056380    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container" containerID="cri-o://86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8" gracePeriod=30
Jun 02 16:17:00.056678 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.056459    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" containerID="cri-o://091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d" gracePeriod=30
Jun 02 16:17:00.056678 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.056529    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-agent" containerID="cri-o://5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a" gracePeriod=30
Jun 02 16:17:00.778006 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.777970    2571 generic.go:358] "Generic (PLEG): container finished" podID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerID="091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d" exitCode=2
Jun 02 16:17:00.778183 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:00.778046    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerDied","Data":"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"}
Jun 02 16:17:02.786533 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:02.786499    2571 generic.go:358] "Generic (PLEG): container finished" podID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerID="86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8" exitCode=0
Jun 02 16:17:02.786937 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:02.786579    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerDied","Data":"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"}
Jun 02 16:17:04.533631 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:04.533589    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:04.540032 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:04.540000    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 02 16:17:09.533054 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:09.533013    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:14.533630 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:14.533583    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:14.534161 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:14.533724    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:17:14.539480 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:14.539454    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 02 16:17:19.532909 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:19.532862    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:24.533106 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:24.533061    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:24.539575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:24.539547    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 02 16:17:24.539682 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:24.539672    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:17:29.533591 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:29.533539    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 02 16:17:30.245583 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.245557    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:17:30.374567 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374467    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location\") pod \"a4bdae6f-4133-462b-a1d9-27c141c600b6\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") "
Jun 02 16:17:30.374567 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374546    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"a4bdae6f-4133-462b-a1d9-27c141c600b6\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") "
Jun 02 16:17:30.374819 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374585    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv42d\" (UniqueName: \"kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d\") pod \"a4bdae6f-4133-462b-a1d9-27c141c600b6\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") "
Jun 02 16:17:30.374819 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374624    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls\") pod \"a4bdae6f-4133-462b-a1d9-27c141c600b6\" (UID: \"a4bdae6f-4133-462b-a1d9-27c141c600b6\") "
Jun 02 16:17:30.374951 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374910    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4bdae6f-4133-462b-a1d9-27c141c600b6" (UID: "a4bdae6f-4133-462b-a1d9-27c141c600b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:30.375006 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.374921    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "a4bdae6f-4133-462b-a1d9-27c141c600b6" (UID: "a4bdae6f-4133-462b-a1d9-27c141c600b6"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:17:30.376780 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.376754    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d" (OuterVolumeSpecName: "kube-api-access-vv42d") pod "a4bdae6f-4133-462b-a1d9-27c141c600b6" (UID: "a4bdae6f-4133-462b-a1d9-27c141c600b6"). InnerVolumeSpecName "kube-api-access-vv42d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:17:30.376873 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.376754    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4bdae6f-4133-462b-a1d9-27c141c600b6" (UID: "a4bdae6f-4133-462b-a1d9-27c141c600b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:17:30.475386 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.475345    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bdae6f-4133-462b-a1d9-27c141c600b6-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:17:30.475566 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.475398    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bdae6f-4133-462b-a1d9-27c141c600b6-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:17:30.475566 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.475416    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv42d\" (UniqueName: \"kubernetes.io/projected/a4bdae6f-4133-462b-a1d9-27c141c600b6-kube-api-access-vv42d\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:17:30.475566 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.475431    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bdae6f-4133-462b-a1d9-27c141c600b6-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:17:30.884475 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.884438    2571 generic.go:358] "Generic (PLEG): container finished" podID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerID="5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a" exitCode=137
Jun 02 16:17:30.884895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.884527    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerDied","Data":"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"}
Jun 02 16:17:30.884895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.884570    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss" event={"ID":"a4bdae6f-4133-462b-a1d9-27c141c600b6","Type":"ContainerDied","Data":"070fd320e4ad73dbea3110c2db1d5bf44e956a91dbdd4679be96e9f9ef75a49f"}
Jun 02 16:17:30.884895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.884587    2571 scope.go:117] "RemoveContainer" containerID="091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"
Jun 02 16:17:30.884895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.884543    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"
Jun 02 16:17:30.893341 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.893325    2571 scope.go:117] "RemoveContainer" containerID="5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"
Jun 02 16:17:30.900476 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.900438    2571 scope.go:117] "RemoveContainer" containerID="86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"
Jun 02 16:17:30.907313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.907296    2571 scope.go:117] "RemoveContainer" containerID="746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"
Jun 02 16:17:30.914224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.914207    2571 scope.go:117] "RemoveContainer" containerID="091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"
Jun 02 16:17:30.914483 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:17:30.914458    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d\": container with ID starting with 091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d not found: ID does not exist" containerID="091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"
Jun 02 16:17:30.914575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.914491    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d"} err="failed to get container status \"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d\": rpc error: code = NotFound desc = could not find container \"091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d\": container with ID starting with 091b05f216ecdb442da2f725dd8f83e9c58db206bc1a77e71e48c96e8fc51a8d not found: ID does not exist"
Jun 02 16:17:30.914575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.914511    2571 scope.go:117] "RemoveContainer" containerID="5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"
Jun 02 16:17:30.914735 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:17:30.914717    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a\": container with ID starting with 5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a not found: ID does not exist" containerID="5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"
Jun 02 16:17:30.914809 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.914739    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a"} err="failed to get container status \"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a\": rpc error: code = NotFound desc = could not find container \"5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a\": container with ID starting with 5382c79b18584a1e8be322b280df799af7d7cef9fab3338658e722333258a71a not found: ID does not exist"
Jun 02 16:17:30.914809 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.914757    2571 scope.go:117] "RemoveContainer" containerID="86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"
Jun 02 16:17:30.915013 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:17:30.914992    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8\": container with ID starting with 86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8 not found: ID does not exist" containerID="86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"
Jun 02 16:17:30.915075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.915016    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8"} err="failed to get container status \"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8\": rpc error: code = NotFound desc = could not find container \"86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8\": container with ID starting with 86c06c5332fda04ad36ffeba9b7cf7fc88100c3e17f233121c42fd628e7968f8 not found: ID does not exist"
Jun 02 16:17:30.915075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.915030    2571 scope.go:117] "RemoveContainer" containerID="746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"
Jun 02 16:17:30.915216 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:17:30.915203    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa\": container with ID starting with 746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa not found: ID does not exist" containerID="746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"
Jun 02 16:17:30.915254 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.915220    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa"} err="failed to get container status \"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa\": rpc error: code = NotFound desc = could not find container \"746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa\": container with ID starting with 746aba0873cfd85d559e0129986d7dc7e9fdec85f3400067de7954cc6d6f94fa not found: ID does not exist"
Jun 02 16:17:30.920870 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.920847    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:17:30.924639 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:30.924618    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c5b7c7699-l4hss"]
Jun 02 16:17:31.684248 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:17:31.684214    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" path="/var/lib/kubelet/pods/a4bdae6f-4133-462b-a1d9-27c141c600b6/volumes"
Jun 02 16:18:21.760204 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760168    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760553    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kserve-container"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760567    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kserve-container"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760576    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760581    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760588    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-agent"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760594    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-agent"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760608    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kube-rbac-proxy"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760613    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kube-rbac-proxy"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760624    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="storage-initializer"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760630    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="storage-initializer"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760637    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760642    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760650    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="storage-initializer"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760655    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="storage-initializer"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760712    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kube-rbac-proxy"
Jun 02 16:18:21.760715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760720    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="50317dae-00ec-4fed-bc11-90f789b217b3" containerName="kserve-container"
Jun 02 16:18:21.761195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760727    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-container"
Jun 02 16:18:21.761195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760735    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kube-rbac-proxy"
Jun 02 16:18:21.761195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.760741    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bdae6f-4133-462b-a1d9-27c141c600b6" containerName="kserve-agent"
Jun 02 16:18:21.764057 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.764034    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.767449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.767423    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:18:21.767449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.767440    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:18:21.767637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.767424    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:18:21.767637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.767427    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:18:21.767637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.767484    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jun 02 16:18:21.775999 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.775967    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:18:21.817975 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.817939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.818141 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.817984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.818141 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.818017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.818141 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.818085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919024 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.918980    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919215 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.919064    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919215 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.919108    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919215 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.919151    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919472 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.919447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.919860 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.919836    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.921509 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.921490    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:21.933839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:21.933815    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:22.075558 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:22.075471    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:22.224784 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:22.224758    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:18:22.226615 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:18:22.226585    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6aa344_9d90_40aa_abfd_ac3c557c1b48.slice/crio-1bb966414908315514ace06413867f432bdc44e0bcf96b3cc1d510febd56de9e WatchSource:0}: Error finding container 1bb966414908315514ace06413867f432bdc44e0bcf96b3cc1d510febd56de9e: Status 404 returned error can't find the container with id 1bb966414908315514ace06413867f432bdc44e0bcf96b3cc1d510febd56de9e
Jun 02 16:18:22.228576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:22.228559    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:18:23.064744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:23.064704    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerStarted","Data":"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"}
Jun 02 16:18:23.064744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:23.064742    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerStarted","Data":"1bb966414908315514ace06413867f432bdc44e0bcf96b3cc1d510febd56de9e"}
Jun 02 16:18:27.083089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:27.082997    2571 generic.go:358] "Generic (PLEG): container finished" podID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerID="7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9" exitCode=0
Jun 02 16:18:27.083089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:27.083058    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerDied","Data":"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"}
Jun 02 16:18:39.128781 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:39.128748    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerStarted","Data":"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"}
Jun 02 16:18:40.135004 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:40.134963    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerStarted","Data":"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"}
Jun 02 16:18:40.135513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:40.135162    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:40.183341 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:40.183288    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podStartSLOduration=7.26474155 podStartE2EDuration="19.183275752s" podCreationTimestamp="2026-06-02 16:18:21 +0000 UTC" firstStartedPulling="2026-06-02 16:18:27.084194325 +0000 UTC m=+1903.959720468" lastFinishedPulling="2026-06-02 16:18:39.002728531 +0000 UTC m=+1915.878254670" observedRunningTime="2026-06-02 16:18:40.181726768 +0000 UTC m=+1917.057252928" watchObservedRunningTime="2026-06-02 16:18:40.183275752 +0000 UTC m=+1917.058801912"
Jun 02 16:18:41.138568 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:41.138524    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:41.140074 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:41.140048    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:18:42.141589 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:42.141543    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:18:47.146062 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:47.146031    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:18:47.146588 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:47.146561    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:18:57.146613 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:18:57.146570    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:19:07.147222 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:07.147180    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:19:17.147415 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:17.147352    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 02 16:19:27.147515 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:27.147479    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:19:33.248286 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:33.248238    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:19:33.248964 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:33.248910    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container" containerID="cri-o://c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e" gracePeriod=30
Jun 02 16:19:33.249973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:33.249204    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kube-rbac-proxy" containerID="cri-o://fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d" gracePeriod=30
Jun 02 16:19:34.327454 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:34.327421    2571 generic.go:358] "Generic (PLEG): container finished" podID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerID="fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d" exitCode=2
Jun 02 16:19:34.327855 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:34.327495    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerDied","Data":"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"}
Jun 02 16:19:36.003080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.003049    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:19:36.179314 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.179216    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") "
Jun 02 16:19:36.179314 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.179275    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location\") pod \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") "
Jun 02 16:19:36.179592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.179351    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw\") pod \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") "
Jun 02 16:19:36.179592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.179404    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls\") pod \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\" (UID: \"6b6aa344-9d90-40aa-abfd-ac3c557c1b48\") "
Jun 02 16:19:36.179727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.179698    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "6b6aa344-9d90-40aa-abfd-ac3c557c1b48" (UID: "6b6aa344-9d90-40aa-abfd-ac3c557c1b48"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:19:36.181496 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.181468    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b6aa344-9d90-40aa-abfd-ac3c557c1b48" (UID: "6b6aa344-9d90-40aa-abfd-ac3c557c1b48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:19:36.181598 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.181553    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw" (OuterVolumeSpecName: "kube-api-access-zmfkw") pod "6b6aa344-9d90-40aa-abfd-ac3c557c1b48" (UID: "6b6aa344-9d90-40aa-abfd-ac3c557c1b48"). InnerVolumeSpecName "kube-api-access-zmfkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:19:36.189158 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.189133    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b6aa344-9d90-40aa-abfd-ac3c557c1b48" (UID: "6b6aa344-9d90-40aa-abfd-ac3c557c1b48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:19:36.280150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.280110    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kube-api-access-zmfkw\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:19:36.280150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.280145    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:19:36.280355 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.280162    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:19:36.280355 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.280177    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b6aa344-9d90-40aa-abfd-ac3c557c1b48-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:19:36.336158 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.336121    2571 generic.go:358] "Generic (PLEG): container finished" podID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerID="c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e" exitCode=0
Jun 02 16:19:36.336330 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.336216    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"
Jun 02 16:19:36.336330 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.336209    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerDied","Data":"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"}
Jun 02 16:19:36.336330 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.336323    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp" event={"ID":"6b6aa344-9d90-40aa-abfd-ac3c557c1b48","Type":"ContainerDied","Data":"1bb966414908315514ace06413867f432bdc44e0bcf96b3cc1d510febd56de9e"}
Jun 02 16:19:36.336469 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.336340    2571 scope.go:117] "RemoveContainer" containerID="fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"
Jun 02 16:19:36.344680 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.344656    2571 scope.go:117] "RemoveContainer" containerID="c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"
Jun 02 16:19:36.352323 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.352302    2571 scope.go:117] "RemoveContainer" containerID="7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"
Jun 02 16:19:36.359888 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.359858    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:19:36.360090 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360055    2571 scope.go:117] "RemoveContainer" containerID="fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"
Jun 02 16:19:36.360332 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:19:36.360310    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d\": container with ID starting with fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d not found: ID does not exist" containerID="fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"
Jun 02 16:19:36.360422 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360339    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d"} err="failed to get container status \"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d\": rpc error: code = NotFound desc = could not find container \"fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d\": container with ID starting with fb9abdc4dd4e3a2988b66f0eda0e7d0adae4ae55e967905d5c9c8839bf066b5d not found: ID does not exist"
Jun 02 16:19:36.360422 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360359    2571 scope.go:117] "RemoveContainer" containerID="c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"
Jun 02 16:19:36.360584 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:19:36.360567    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e\": container with ID starting with c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e not found: ID does not exist" containerID="c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"
Jun 02 16:19:36.360644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360592    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e"} err="failed to get container status \"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e\": rpc error: code = NotFound desc = could not find container \"c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e\": container with ID starting with c4041a7ce291dcf5e857d502f8556209077be8a2dfd10fb0c92545b523b7eb8e not found: ID does not exist"
Jun 02 16:19:36.360644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360615    2571 scope.go:117] "RemoveContainer" containerID="7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"
Jun 02 16:19:36.360827 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:19:36.360809    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9\": container with ID starting with 7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9 not found: ID does not exist" containerID="7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"
Jun 02 16:19:36.360868 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.360831    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9"} err="failed to get container status \"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9\": rpc error: code = NotFound desc = could not find container \"7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9\": container with ID starting with 7a7a4f122204800c027cff6d66b686f042757b0ee05d19796ca69a52affae1d9 not found: ID does not exist"
Jun 02 16:19:36.364147 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:36.364120    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9svkp"]
Jun 02 16:19:37.685459 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:19:37.685418    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" path="/var/lib/kubelet/pods/6b6aa344-9d90-40aa-abfd-ac3c557c1b48/volumes"
Jun 02 16:21:43.749742 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:21:43.749714    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:21:43.752143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:21:43.752122    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:25:38.765457 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765412    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="storage-initializer"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765837    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="storage-initializer"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765853    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kube-rbac-proxy"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765858    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kube-rbac-proxy"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765869    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765875    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765942    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kube-rbac-proxy"
Jun 02 16:25:38.766011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.765957    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6b6aa344-9d90-40aa-abfd-ac3c557c1b48" containerName="kserve-container"
Jun 02 16:25:38.768003 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.767987    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.770196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.770177    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:25:38.771200 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.771173    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-271d24-predictor-serving-cert\""
Jun 02 16:25:38.771200 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.771194    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-271d24-kube-rbac-proxy-sar-config\""
Jun 02 16:25:38.771401 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.771323    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:25:38.771401 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.771354    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:25:38.779301 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.779278    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:25:38.823783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.823743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p682\" (UniqueName: \"kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.823783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.823782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.823978 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.823815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.823978 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.823871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.924693 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.924652    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5p682\" (UniqueName: \"kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.924693 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.924699    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.924950 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.924736    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.924950 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.924767    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.925276 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.925251    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.925603 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.925584    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.927432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.927411    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:38.935453 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:38.935428    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p682\" (UniqueName: \"kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682\") pod \"isvc-primary-271d24-predictor-6c7769446c-5gzcx\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") " pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:39.079080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:39.078989    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:39.212107 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:39.212040    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:25:39.214667 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:25:39.214638    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b22614_f323_45bb_b75f_9a1380b51c4e.slice/crio-c15237c15242c2ff20d476919bb790e8011eb87471cca09b3437097709288ab6 WatchSource:0}: Error finding container c15237c15242c2ff20d476919bb790e8011eb87471cca09b3437097709288ab6: Status 404 returned error can't find the container with id c15237c15242c2ff20d476919bb790e8011eb87471cca09b3437097709288ab6
Jun 02 16:25:39.216902 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:39.216885    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:25:39.561350 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:39.561308    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerStarted","Data":"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"}
Jun 02 16:25:39.561350 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:39.561348    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerStarted","Data":"c15237c15242c2ff20d476919bb790e8011eb87471cca09b3437097709288ab6"}
Jun 02 16:25:43.575493 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:43.575460    2571 generic.go:358] "Generic (PLEG): container finished" podID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerID="bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025" exitCode=0
Jun 02 16:25:43.575880 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:43.575532    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerDied","Data":"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"}
Jun 02 16:25:44.580661 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.580625    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerStarted","Data":"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"}
Jun 02 16:25:44.580661 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.580665    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerStarted","Data":"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"}
Jun 02 16:25:44.581156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.580981    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:44.581156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.581122    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:44.582506 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.582483    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:25:44.598830 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:44.598782    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podStartSLOduration=6.5987666449999995 podStartE2EDuration="6.598766645s" podCreationTimestamp="2026-06-02 16:25:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:25:44.597554947 +0000 UTC m=+2341.473081108" watchObservedRunningTime="2026-06-02 16:25:44.598766645 +0000 UTC m=+2341.474292784"
Jun 02 16:25:45.585126 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:45.585085    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:25:46.588698 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:46.588658    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:25:51.593712 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:51.593680    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:25:51.594255 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:25:51.594230    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:01.594943 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:01.594899    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:11.594212 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:11.594167    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:21.594839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:21.594781    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:31.595046 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:31.595003    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:41.594752 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:41.594710    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 02 16:26:43.774019 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:43.773988    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:26:43.777879 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:43.777860    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:26:51.595546 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:51.595459    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:26:58.889815 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.889773    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:26:58.892521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.892500    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:58.894799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.894776    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-271d24\""
Jun 02 16:26:58.894921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.894780    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 02 16:26:58.894921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.894784    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-271d24-predictor-serving-cert\""
Jun 02 16:26:58.895092 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.895079    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-271d24-kube-rbac-proxy-sar-config\""
Jun 02 16:26:58.895489 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.895472    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-271d24-dockercfg-5ssbl\""
Jun 02 16:26:58.903030 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:58.903007    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:26:59.025546 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.025508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.025734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.025609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.025734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.025648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqn2c\" (UniqueName: \"kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.025734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.025668    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.025734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.025692    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.126648    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.126690    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zqn2c\" (UniqueName: \"kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126961 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.126709    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126961 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.126730    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126961 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.126785    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.126961 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:26:59.126893    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-271d24-predictor-serving-cert: secret "isvc-secondary-271d24-predictor-serving-cert" not found
Jun 02 16:26:59.127181 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:26:59.126984    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls podName:35eac83a-a406-4821-891b-99da655dd367 nodeName:}" failed. No retries permitted until 2026-06-02 16:26:59.626948976 +0000 UTC m=+2416.502475130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls") pod "isvc-secondary-271d24-predictor-5598b65d68-kmw7f" (UID: "35eac83a-a406-4821-891b-99da655dd367") : secret "isvc-secondary-271d24-predictor-serving-cert" not found
Jun 02 16:26:59.127251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.127177    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.127449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.127427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.127536 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.127468    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.138010 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.137988    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqn2c\" (UniqueName: \"kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.631644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.631601    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.634165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.634141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") " pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.803954 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.803920    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:26:59.934156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:26:59.934126    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:26:59.937044 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:26:59.937008    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eac83a_a406_4821_891b_99da655dd367.slice/crio-5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502 WatchSource:0}: Error finding container 5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502: Status 404 returned error can't find the container with id 5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502
Jun 02 16:27:00.840592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:00.840556    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerStarted","Data":"481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604"}
Jun 02 16:27:00.840769 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:00.840599    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerStarted","Data":"5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502"}
Jun 02 16:27:03.851321 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:03.851292    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/0.log"
Jun 02 16:27:03.851744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:03.851336    2571 generic.go:358] "Generic (PLEG): container finished" podID="35eac83a-a406-4821-891b-99da655dd367" containerID="481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604" exitCode=1
Jun 02 16:27:03.851744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:03.851422    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerDied","Data":"481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604"}
Jun 02 16:27:04.855992 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:04.855961    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/0.log"
Jun 02 16:27:04.856415 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:04.856076    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerStarted","Data":"b3c6de332693787475d8c6ea254b953c87e15a2212cbbfbc752c99bb51b7b7ad"}
Jun 02 16:27:07.868173 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.868142    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/1.log"
Jun 02 16:27:07.868602 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.868530    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/0.log"
Jun 02 16:27:07.868602 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.868562    2571 generic.go:358] "Generic (PLEG): container finished" podID="35eac83a-a406-4821-891b-99da655dd367" containerID="b3c6de332693787475d8c6ea254b953c87e15a2212cbbfbc752c99bb51b7b7ad" exitCode=1
Jun 02 16:27:07.868682 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.868634    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerDied","Data":"b3c6de332693787475d8c6ea254b953c87e15a2212cbbfbc752c99bb51b7b7ad"}
Jun 02 16:27:07.868682 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.868673    2571 scope.go:117] "RemoveContainer" containerID="481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604"
Jun 02 16:27:07.869068 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:07.869052    2571 scope.go:117] "RemoveContainer" containerID="481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604"
Jun 02 16:27:07.880125 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:07.880093    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_kserve-ci-e2e-test_35eac83a-a406-4821-891b-99da655dd367_0 in pod sandbox 5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502 from index: no such id: '481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604'" containerID="481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604"
Jun 02 16:27:07.880202 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:07.880145    2571 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_kserve-ci-e2e-test_35eac83a-a406-4821-891b-99da655dd367_0 in pod sandbox 5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502 from index: no such id: '481d60835c65d33f01984d036e08bd62e9bdb76f141ba89e166271520ef8c604'; Skipping pod \"isvc-secondary-271d24-predictor-5598b65d68-kmw7f_kserve-ci-e2e-test(35eac83a-a406-4821-891b-99da655dd367)\"" logger="UnhandledError"
Jun 02 16:27:07.881561 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:07.881538    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-271d24-predictor-5598b65d68-kmw7f_kserve-ci-e2e-test(35eac83a-a406-4821-891b-99da655dd367)\"" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" podUID="35eac83a-a406-4821-891b-99da655dd367"
Jun 02 16:27:08.873696 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:08.873671    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/1.log"
Jun 02 16:27:14.919752 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:14.919713    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:27:14.986507 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:14.986471    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:27:14.986961 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:14.986911    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container" containerID="cri-o://9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f" gracePeriod=30
Jun 02 16:27:14.987191 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:14.986988    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kube-rbac-proxy" containerID="cri-o://36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96" gracePeriod=30
Jun 02 16:27:15.097973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.097938    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:15.102875 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.102848    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.105078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.105053    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b0ca37-predictor-serving-cert\""
Jun 02 16:27:15.105078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.105071    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\""
Jun 02 16:27:15.105078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.105057    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b0ca37-dockercfg-8n5rq\""
Jun 02 16:27:15.105380 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.105353    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b0ca37\""
Jun 02 16:27:15.111225 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.111197    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:15.114272 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.114253    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/1.log"
Jun 02 16:27:15.114400 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.114325    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:27:15.169075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169042    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert\") pod \"35eac83a-a406-4821-891b-99da655dd367\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") "
Jun 02 16:27:15.169264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169089    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location\") pod \"35eac83a-a406-4821-891b-99da655dd367\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") "
Jun 02 16:27:15.169264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169118    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqn2c\" (UniqueName: \"kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c\") pod \"35eac83a-a406-4821-891b-99da655dd367\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") "
Jun 02 16:27:15.169264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169166    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") pod \"35eac83a-a406-4821-891b-99da655dd367\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") "
Jun 02 16:27:15.169264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169206    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config\") pod \"35eac83a-a406-4821-891b-99da655dd367\" (UID: \"35eac83a-a406-4821-891b-99da655dd367\") "
Jun 02 16:27:15.169513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.169513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdpm\" (UniqueName: \"kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.169513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169446    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35eac83a-a406-4821-891b-99da655dd367" (UID: "35eac83a-a406-4821-891b-99da655dd367"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:27:15.169513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.169513 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169501    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.169727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.169727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169486    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35eac83a-a406-4821-891b-99da655dd367" (UID: "35eac83a-a406-4821-891b-99da655dd367"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:15.169727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169651    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-271d24-kube-rbac-proxy-sar-config") pod "35eac83a-a406-4821-891b-99da655dd367" (UID: "35eac83a-a406-4821-891b-99da655dd367"). InnerVolumeSpecName "isvc-secondary-271d24-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:15.169727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169721    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35eac83a-a406-4821-891b-99da655dd367-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:15.169886 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169735    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-isvc-secondary-271d24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:15.169886 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.169746    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35eac83a-a406-4821-891b-99da655dd367-cabundle-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:15.171411 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.171393    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c" (OuterVolumeSpecName: "kube-api-access-zqn2c") pod "35eac83a-a406-4821-891b-99da655dd367" (UID: "35eac83a-a406-4821-891b-99da655dd367"). InnerVolumeSpecName "kube-api-access-zqn2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:27:15.171411 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.171403    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35eac83a-a406-4821-891b-99da655dd367" (UID: "35eac83a-a406-4821-891b-99da655dd367"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:27:15.270607 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270567    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdpm\" (UniqueName: \"kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.270811 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270615    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.270811 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270643    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.270811 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270699    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.270993 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270818    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.270993 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270866    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqn2c\" (UniqueName: \"kubernetes.io/projected/35eac83a-a406-4821-891b-99da655dd367-kube-api-access-zqn2c\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:15.270993 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.270885    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35eac83a-a406-4821-891b-99da655dd367-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:15.271254 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.271234    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.271357 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.271337    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.271428 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.271344    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.273158 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.273138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.278207 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.278189    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdpm\" (UniqueName: \"kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm\") pod \"isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") " pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.424234 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.424129    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:15.556334 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.556306    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:15.558522 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:27:15.558492    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661573c0_2192_4977_a774_8de4128ff3ed.slice/crio-f3b6463b4a9ae31f9df81c1197475f5c40014b28080681a590eee09bb80e140c WatchSource:0}: Error finding container f3b6463b4a9ae31f9df81c1197475f5c40014b28080681a590eee09bb80e140c: Status 404 returned error can't find the container with id f3b6463b4a9ae31f9df81c1197475f5c40014b28080681a590eee09bb80e140c
Jun 02 16:27:15.898545 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.898516    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-271d24-predictor-5598b65d68-kmw7f_35eac83a-a406-4821-891b-99da655dd367/storage-initializer/1.log"
Jun 02 16:27:15.898723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.898616    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f" event={"ID":"35eac83a-a406-4821-891b-99da655dd367","Type":"ContainerDied","Data":"5832f07453fbb3e2376e1c936aaa5ff7c0c1fc498e13a2531e2e9e357d48e502"}
Jun 02 16:27:15.898723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.898657    2571 scope.go:117] "RemoveContainer" containerID="b3c6de332693787475d8c6ea254b953c87e15a2212cbbfbc752c99bb51b7b7ad"
Jun 02 16:27:15.898723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.898679    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"
Jun 02 16:27:15.900770 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.900738    2571 generic.go:358] "Generic (PLEG): container finished" podID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerID="36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96" exitCode=2
Jun 02 16:27:15.900912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.900813    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerDied","Data":"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"}
Jun 02 16:27:15.902592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.902570    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerStarted","Data":"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"}
Jun 02 16:27:15.902687 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.902602    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerStarted","Data":"f3b6463b4a9ae31f9df81c1197475f5c40014b28080681a590eee09bb80e140c"}
Jun 02 16:27:15.929605 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.929567    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:27:15.932476 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:15.932451    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-271d24-predictor-5598b65d68-kmw7f"]
Jun 02 16:27:16.589472 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:16.589432    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jun 02 16:27:17.684203 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:17.684167    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35eac83a-a406-4821-891b-99da655dd367" path="/var/lib/kubelet/pods/35eac83a-a406-4821-891b-99da655dd367/volumes"
Jun 02 16:27:18.915634 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:18.915600    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/0.log"
Jun 02 16:27:18.916043 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:18.915642    2571 generic.go:358] "Generic (PLEG): container finished" podID="661573c0-2192-4977-a774-8de4128ff3ed" containerID="55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629" exitCode=1
Jun 02 16:27:18.916043 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:18.915721    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerDied","Data":"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"}
Jun 02 16:27:19.434015 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.433986    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:27:19.508756 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.508656    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p682\" (UniqueName: \"kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682\") pod \"24b22614-f323-45bb-b75f-9a1380b51c4e\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") "
Jun 02 16:27:19.508756 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.508746    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location\") pod \"24b22614-f323-45bb-b75f-9a1380b51c4e\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") "
Jun 02 16:27:19.509004 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.508807    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls\") pod \"24b22614-f323-45bb-b75f-9a1380b51c4e\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") "
Jun 02 16:27:19.509004 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.508861    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config\") pod \"24b22614-f323-45bb-b75f-9a1380b51c4e\" (UID: \"24b22614-f323-45bb-b75f-9a1380b51c4e\") "
Jun 02 16:27:19.509123 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.509092    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24b22614-f323-45bb-b75f-9a1380b51c4e" (UID: "24b22614-f323-45bb-b75f-9a1380b51c4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:27:19.509202 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.509184    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-271d24-kube-rbac-proxy-sar-config") pod "24b22614-f323-45bb-b75f-9a1380b51c4e" (UID: "24b22614-f323-45bb-b75f-9a1380b51c4e"). InnerVolumeSpecName "isvc-primary-271d24-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:19.510827 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.510800    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24b22614-f323-45bb-b75f-9a1380b51c4e" (UID: "24b22614-f323-45bb-b75f-9a1380b51c4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:27:19.511224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.511209    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682" (OuterVolumeSpecName: "kube-api-access-5p682") pod "24b22614-f323-45bb-b75f-9a1380b51c4e" (UID: "24b22614-f323-45bb-b75f-9a1380b51c4e"). InnerVolumeSpecName "kube-api-access-5p682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:27:19.609659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.609621    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b22614-f323-45bb-b75f-9a1380b51c4e-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.609659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.609652    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b22614-f323-45bb-b75f-9a1380b51c4e-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.609659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.609665    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-271d24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b22614-f323-45bb-b75f-9a1380b51c4e-isvc-primary-271d24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.609887 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.609675    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p682\" (UniqueName: \"kubernetes.io/projected/24b22614-f323-45bb-b75f-9a1380b51c4e-kube-api-access-5p682\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.920427 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.920399    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/0.log"
Jun 02 16:27:19.920853 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.920545    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerStarted","Data":"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"}
Jun 02 16:27:19.922358 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.922326    2571 generic.go:358] "Generic (PLEG): container finished" podID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerID="9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f" exitCode=0
Jun 02 16:27:19.922485 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.922401    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerDied","Data":"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"}
Jun 02 16:27:19.922485 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.922429    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx" event={"ID":"24b22614-f323-45bb-b75f-9a1380b51c4e","Type":"ContainerDied","Data":"c15237c15242c2ff20d476919bb790e8011eb87471cca09b3437097709288ab6"}
Jun 02 16:27:19.922485 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.922430    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"
Jun 02 16:27:19.922485 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.922453    2571 scope.go:117] "RemoveContainer" containerID="36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"
Jun 02 16:27:19.931023 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.931001    2571 scope.go:117] "RemoveContainer" containerID="9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"
Jun 02 16:27:19.939820 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.939802    2571 scope.go:117] "RemoveContainer" containerID="bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"
Jun 02 16:27:19.947576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.947551    2571 scope.go:117] "RemoveContainer" containerID="36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"
Jun 02 16:27:19.947900 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:19.947883    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96\": container with ID starting with 36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96 not found: ID does not exist" containerID="36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"
Jun 02 16:27:19.947965 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.947907    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96"} err="failed to get container status \"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96\": rpc error: code = NotFound desc = could not find container \"36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96\": container with ID starting with 36cddef348e3a3956b476df6d207b1215939393b1a1c440a12b79aa653a8ac96 not found: ID does not exist"
Jun 02 16:27:19.947965 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.947925    2571 scope.go:117] "RemoveContainer" containerID="9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"
Jun 02 16:27:19.948164 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:19.948148    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f\": container with ID starting with 9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f not found: ID does not exist" containerID="9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"
Jun 02 16:27:19.948205 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.948170    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f"} err="failed to get container status \"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f\": rpc error: code = NotFound desc = could not find container \"9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f\": container with ID starting with 9263b8a047652e9554fb6ad918e2c978063188285824c29b81355c0736f3234f not found: ID does not exist"
Jun 02 16:27:19.948205 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.948182    2571 scope.go:117] "RemoveContainer" containerID="bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"
Jun 02 16:27:19.948417 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:19.948401    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025\": container with ID starting with bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025 not found: ID does not exist" containerID="bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"
Jun 02 16:27:19.948469 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.948419    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025"} err="failed to get container status \"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025\": rpc error: code = NotFound desc = could not find container \"bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025\": container with ID starting with bf23cc3074bb381995dd4f871b5dec60711202a763643ce5432142325ba26025 not found: ID does not exist"
Jun 02 16:27:19.952441 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.952418    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:27:19.958807 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:19.958783    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-271d24-predictor-6c7769446c-5gzcx"]
Jun 02 16:27:20.049689 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:20.049657    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:20.927598 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:20.927551    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer" containerID="cri-o://3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a" gracePeriod=30
Jun 02 16:27:21.684131 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.684091    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" path="/var/lib/kubelet/pods/24b22614-f323-45bb-b75f-9a1380b51c4e/volumes"
Jun 02 16:27:21.877398 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.877356    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/1.log"
Jun 02 16:27:21.877774 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.877757    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/0.log"
Jun 02 16:27:21.877941 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.877841    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:21.932135 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932056    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/1.log"
Jun 02 16:27:21.932530 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932515    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2_661573c0-2192-4977-a774-8de4128ff3ed/storage-initializer/0.log"
Jun 02 16:27:21.932598 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932548    2571 generic.go:358] "Generic (PLEG): container finished" podID="661573c0-2192-4977-a774-8de4128ff3ed" containerID="3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a" exitCode=1
Jun 02 16:27:21.932654 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932609    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerDied","Data":"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"}
Jun 02 16:27:21.932654 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932632    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2" event={"ID":"661573c0-2192-4977-a774-8de4128ff3ed","Type":"ContainerDied","Data":"f3b6463b4a9ae31f9df81c1197475f5c40014b28080681a590eee09bb80e140c"}
Jun 02 16:27:21.932654 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932646    2571 scope.go:117] "RemoveContainer" containerID="3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"
Jun 02 16:27:21.932843 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.932663    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"
Jun 02 16:27:21.937440 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937420    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls\") pod \"661573c0-2192-4977-a774-8de4128ff3ed\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") "
Jun 02 16:27:21.937555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937457    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location\") pod \"661573c0-2192-4977-a774-8de4128ff3ed\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") "
Jun 02 16:27:21.937555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937501    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgdpm\" (UniqueName: \"kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm\") pod \"661573c0-2192-4977-a774-8de4128ff3ed\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") "
Jun 02 16:27:21.937555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937526    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\") pod \"661573c0-2192-4977-a774-8de4128ff3ed\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") "
Jun 02 16:27:21.937701 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937556    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert\") pod \"661573c0-2192-4977-a774-8de4128ff3ed\" (UID: \"661573c0-2192-4977-a774-8de4128ff3ed\") "
Jun 02 16:27:21.937925 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937900    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "661573c0-2192-4977-a774-8de4128ff3ed" (UID: "661573c0-2192-4977-a774-8de4128ff3ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:27:21.938011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937991    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config") pod "661573c0-2192-4977-a774-8de4128ff3ed" (UID: "661573c0-2192-4977-a774-8de4128ff3ed"). InnerVolumeSpecName "isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:21.938011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.937998    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "661573c0-2192-4977-a774-8de4128ff3ed" (UID: "661573c0-2192-4977-a774-8de4128ff3ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:21.939708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.939678    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "661573c0-2192-4977-a774-8de4128ff3ed" (UID: "661573c0-2192-4977-a774-8de4128ff3ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:27:21.939708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.939677    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm" (OuterVolumeSpecName: "kube-api-access-mgdpm") pod "661573c0-2192-4977-a774-8de4128ff3ed" (UID: "661573c0-2192-4977-a774-8de4128ff3ed"). InnerVolumeSpecName "kube-api-access-mgdpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:27:21.941089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.941072    2571 scope.go:117] "RemoveContainer" containerID="55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"
Jun 02 16:27:21.957694 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.957668    2571 scope.go:117] "RemoveContainer" containerID="3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"
Jun 02 16:27:21.958258 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:21.958234    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a\": container with ID starting with 3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a not found: ID does not exist" containerID="3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"
Jun 02 16:27:21.958342 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.958271    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a"} err="failed to get container status \"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a\": rpc error: code = NotFound desc = could not find container \"3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a\": container with ID starting with 3f312bed21184b396fc91c4290c9489cbd32c3bd13037d268054b130abf9bf7a not found: ID does not exist"
Jun 02 16:27:21.958342 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.958290    2571 scope.go:117] "RemoveContainer" containerID="55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"
Jun 02 16:27:21.958643 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:27:21.958627    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629\": container with ID starting with 55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629 not found: ID does not exist" containerID="55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"
Jun 02 16:27:21.958690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:21.958646    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629"} err="failed to get container status \"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629\": rpc error: code = NotFound desc = could not find container \"55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629\": container with ID starting with 55a88113188bd91ae0bd356a6c53a81d699db22813710b1e8f1c9903890c5629 not found: ID does not exist"
Jun 02 16:27:22.038737 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.038697    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdpm\" (UniqueName: \"kubernetes.io/projected/661573c0-2192-4977-a774-8de4128ff3ed-kube-api-access-mgdpm\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:22.038737 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.038731    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-isvc-init-fail-b0ca37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:22.038737 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.038743    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/661573c0-2192-4977-a774-8de4128ff3ed-cabundle-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:22.038974 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.038752    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/661573c0-2192-4977-a774-8de4128ff3ed-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:22.038974 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.038761    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/661573c0-2192-4977-a774-8de4128ff3ed-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:27:22.269883 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.269844    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:22.274702 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:22.274674    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b0ca37-predictor-59dcb8bc4d-kzdc2"]
Jun 02 16:27:23.685645 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:27:23.685611    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661573c0-2192-4977-a774-8de4128ff3ed" path="/var/lib/kubelet/pods/661573c0-2192-4977-a774-8de4128ff3ed/volumes"
Jun 02 16:31:43.804947 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:31:43.804914    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:31:43.810455 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:31:43.810434    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:36:43.833594 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.833566    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:36:43.837851 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.837832    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:36:43.850329 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850303    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:36:43.850668 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850656    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kube-rbac-proxy"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850670    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kube-rbac-proxy"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850681    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850687    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850694    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="storage-initializer"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850699    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="storage-initializer"
Jun 02 16:36:43.850711 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850713    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850718    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850731    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850738    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850744    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850750    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850804    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850821    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kube-rbac-proxy"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850828    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850836    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="24b22614-f323-45bb-b75f-9a1380b51c4e" containerName="kserve-container"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850843    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850850    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="661573c0-2192-4977-a774-8de4128ff3ed" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850899    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.850904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.850904    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="35eac83a-a406-4821-891b-99da655dd367" containerName="storage-initializer"
Jun 02 16:36:43.854055 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.854036    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:43.856172 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.856150    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jun 02 16:36:43.856313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.856206    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jun 02 16:36:43.856313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.856243    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:36:43.856520 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.856501    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:36:43.857070 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.857050    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:36:43.864918 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.864893    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:36:43.962477 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.962437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:43.962659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.962493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7hl\" (UniqueName: \"kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:43.962659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.962515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:43.962659 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:43.962553    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.063597 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.063560    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.063808 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.063615    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7hl\" (UniqueName: \"kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.063808 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.063639    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.063808 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:36:44.063736    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Jun 02 16:36:44.063808 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.063765    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.063808 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:36:44.063802    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls podName:bb732b68-9580-4713-96b7-404487c7330c nodeName:}" failed. No retries permitted until 2026-06-02 16:36:44.563782448 +0000 UTC m=+3001.439308588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls") pod "isvc-sklearn-predictor-55dfdc8df6-x26jd" (UID: "bb732b68-9580-4713-96b7-404487c7330c") : secret "isvc-sklearn-predictor-serving-cert" not found
Jun 02 16:36:44.064079 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.064064    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.064301 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.064283    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.073747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.073728    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7hl\" (UniqueName: \"kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.568241 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.568193    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.570647 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.570626    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") pod \"isvc-sklearn-predictor-55dfdc8df6-x26jd\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.764925 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.764879    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:44.899567 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.899532    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:36:44.903255 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:36:44.903228    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb732b68_9580_4713_96b7_404487c7330c.slice/crio-c294178282ff3bfaf0ac09b45c09372b8df32822b65b52c5904b07c1d2c56ff1 WatchSource:0}: Error finding container c294178282ff3bfaf0ac09b45c09372b8df32822b65b52c5904b07c1d2c56ff1: Status 404 returned error can't find the container with id c294178282ff3bfaf0ac09b45c09372b8df32822b65b52c5904b07c1d2c56ff1
Jun 02 16:36:44.905196 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:44.905175    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:36:45.867980 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:45.867946    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerStarted","Data":"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"}
Jun 02 16:36:45.867980 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:45.867979    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerStarted","Data":"c294178282ff3bfaf0ac09b45c09372b8df32822b65b52c5904b07c1d2c56ff1"}
Jun 02 16:36:48.880581 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:48.880548    2571 generic.go:358] "Generic (PLEG): container finished" podID="bb732b68-9580-4713-96b7-404487c7330c" containerID="d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352" exitCode=0
Jun 02 16:36:48.880973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:48.880622    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerDied","Data":"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"}
Jun 02 16:36:49.885589 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:49.885554    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerStarted","Data":"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"}
Jun 02 16:36:49.885589 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:49.885591    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerStarted","Data":"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"}
Jun 02 16:36:49.886007 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:49.885801    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:49.904753 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:49.904701    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podStartSLOduration=6.904685317 podStartE2EDuration="6.904685317s" podCreationTimestamp="2026-06-02 16:36:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:36:49.903455794 +0000 UTC m=+3006.778981955" watchObservedRunningTime="2026-06-02 16:36:49.904685317 +0000 UTC m=+3006.780211477"
Jun 02 16:36:50.888849 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:50.888809    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:50.889828 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:50.889800    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:36:51.892323 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:51.892283    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:36:56.897121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:56.897090    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:36:56.897634 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:36:56.897608    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:06.897882 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:06.897843    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:16.897633 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:16.897591    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:26.897861 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:26.897817    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:36.898304 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:36.898258    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:46.898229 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:46.898187    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:37:56.898527 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:37:56.898492    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:38:03.971523 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:03.971488    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:38:03.971938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:03.971822    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" containerID="cri-o://e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82" gracePeriod=30
Jun 02 16:38:03.971938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:03.971876    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kube-rbac-proxy" containerID="cri-o://d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320" gracePeriod=30
Jun 02 16:38:04.056873 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.056839    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:38:04.060541 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.060523    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.062834 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.062806    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jun 02 16:38:04.062966 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.062849    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 02 16:38:04.069225 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.069200    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:38:04.143938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.143905    2571 generic.go:358] "Generic (PLEG): container finished" podID="bb732b68-9580-4713-96b7-404487c7330c" containerID="d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320" exitCode=2
Jun 02 16:38:04.143938 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.143942    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerDied","Data":"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"}
Jun 02 16:38:04.171604 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.171571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqd4\" (UniqueName: \"kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.171772 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.171616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.171772 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.171635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.171772 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.171712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.272969 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.272880    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqd4\" (UniqueName: \"kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.272969 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.272932    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.273207 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.272976    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.273207 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.273024    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.273531 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.273507    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.273783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.273763    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.275467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.275448    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.281148 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.281122    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqd4\" (UniqueName: \"kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-2wgp7\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.371771 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.371735    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:04.506521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:04.506487    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:38:04.509744 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:38:04.509713    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b361932_3069_4ff6_b0df_311003ba670e.slice/crio-28b505b3dbfb9a38d6f87291a44c126697b2569b87f7028beba9fb5c176e1aff WatchSource:0}: Error finding container 28b505b3dbfb9a38d6f87291a44c126697b2569b87f7028beba9fb5c176e1aff: Status 404 returned error can't find the container with id 28b505b3dbfb9a38d6f87291a44c126697b2569b87f7028beba9fb5c176e1aff
Jun 02 16:38:05.148402 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:05.148343    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerStarted","Data":"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"}
Jun 02 16:38:05.148402 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:05.148404    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerStarted","Data":"28b505b3dbfb9a38d6f87291a44c126697b2569b87f7028beba9fb5c176e1aff"}
Jun 02 16:38:06.893394 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:06.893326    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jun 02 16:38:06.897685 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:06.897654    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jun 02 16:38:08.524176 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.524147    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:38:08.607017 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.606927    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq7hl\" (UniqueName: \"kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl\") pod \"bb732b68-9580-4713-96b7-404487c7330c\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") "
Jun 02 16:38:08.607017 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.606989    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"bb732b68-9580-4713-96b7-404487c7330c\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") "
Jun 02 16:38:08.607260 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.607022    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") pod \"bb732b68-9580-4713-96b7-404487c7330c\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") "
Jun 02 16:38:08.607260 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.607065    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location\") pod \"bb732b68-9580-4713-96b7-404487c7330c\" (UID: \"bb732b68-9580-4713-96b7-404487c7330c\") "
Jun 02 16:38:08.607432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.607407    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "bb732b68-9580-4713-96b7-404487c7330c" (UID: "bb732b68-9580-4713-96b7-404487c7330c"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:38:08.607493 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.607453    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb732b68-9580-4713-96b7-404487c7330c" (UID: "bb732b68-9580-4713-96b7-404487c7330c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:38:08.609100 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.609079    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb732b68-9580-4713-96b7-404487c7330c" (UID: "bb732b68-9580-4713-96b7-404487c7330c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:38:08.609206 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.609184    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl" (OuterVolumeSpecName: "kube-api-access-qq7hl") pod "bb732b68-9580-4713-96b7-404487c7330c" (UID: "bb732b68-9580-4713-96b7-404487c7330c"). InnerVolumeSpecName "kube-api-access-qq7hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:38:08.707641 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.707604    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb732b68-9580-4713-96b7-404487c7330c-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:38:08.707641 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.707636    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7hl\" (UniqueName: \"kubernetes.io/projected/bb732b68-9580-4713-96b7-404487c7330c-kube-api-access-qq7hl\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:38:08.707641 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.707646    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb732b68-9580-4713-96b7-404487c7330c-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:38:08.707888 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:08.707658    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb732b68-9580-4713-96b7-404487c7330c-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:38:09.164019 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.163982    2571 generic.go:358] "Generic (PLEG): container finished" podID="bb732b68-9580-4713-96b7-404487c7330c" containerID="e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82" exitCode=0
Jun 02 16:38:09.164223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.164067    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"
Jun 02 16:38:09.164223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.164063    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerDied","Data":"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"}
Jun 02 16:38:09.164223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.164170    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd" event={"ID":"bb732b68-9580-4713-96b7-404487c7330c","Type":"ContainerDied","Data":"c294178282ff3bfaf0ac09b45c09372b8df32822b65b52c5904b07c1d2c56ff1"}
Jun 02 16:38:09.164223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.164189    2571 scope.go:117] "RemoveContainer" containerID="d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"
Jun 02 16:38:09.165553 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.165527    2571 generic.go:358] "Generic (PLEG): container finished" podID="2b361932-3069-4ff6-b0df-311003ba670e" containerID="cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3" exitCode=0
Jun 02 16:38:09.165553 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.165564    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerDied","Data":"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"}
Jun 02 16:38:09.173019 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.173002    2571 scope.go:117] "RemoveContainer" containerID="e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"
Jun 02 16:38:09.181822 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.181796    2571 scope.go:117] "RemoveContainer" containerID="d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"
Jun 02 16:38:09.190775 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.190716    2571 scope.go:117] "RemoveContainer" containerID="d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"
Jun 02 16:38:09.191109 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:38:09.191080    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320\": container with ID starting with d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320 not found: ID does not exist" containerID="d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"
Jun 02 16:38:09.191239 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.191121    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320"} err="failed to get container status \"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320\": rpc error: code = NotFound desc = could not find container \"d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320\": container with ID starting with d6aeb800a835d4a9db4727023559fd8a43d0759cb7119135bd2258bfa0ff3320 not found: ID does not exist"
Jun 02 16:38:09.191239 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.191148    2571 scope.go:117] "RemoveContainer" containerID="e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"
Jun 02 16:38:09.191726 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:38:09.191433    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82\": container with ID starting with e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82 not found: ID does not exist" containerID="e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"
Jun 02 16:38:09.191726 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.191465    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82"} err="failed to get container status \"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82\": rpc error: code = NotFound desc = could not find container \"e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82\": container with ID starting with e2d4d9b72192c0e15d3e3ff4575d72b6b7247b12c409aa046bf6e1528a2eae82 not found: ID does not exist"
Jun 02 16:38:09.191726 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.191481    2571 scope.go:117] "RemoveContainer" containerID="d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"
Jun 02 16:38:09.191908 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:38:09.191752    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352\": container with ID starting with d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352 not found: ID does not exist" containerID="d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"
Jun 02 16:38:09.191908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.191781    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352"} err="failed to get container status \"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352\": rpc error: code = NotFound desc = could not find container \"d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352\": container with ID starting with d9f5206613fd9ba5f55cfe75b85af67f1c57dfdea2885ec0800d7b8361bd9352 not found: ID does not exist"
Jun 02 16:38:09.201884 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.201855    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:38:09.206098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.206070    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-55dfdc8df6-x26jd"]
Jun 02 16:38:09.684862 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:09.684830    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb732b68-9580-4713-96b7-404487c7330c" path="/var/lib/kubelet/pods/bb732b68-9580-4713-96b7-404487c7330c/volumes"
Jun 02 16:38:10.171986 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:10.171951    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerStarted","Data":"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"}
Jun 02 16:38:10.172178 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:10.171994    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerStarted","Data":"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"}
Jun 02 16:38:10.172267 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:10.172243    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:10.194504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:10.194456    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" podStartSLOduration=6.194438446 podStartE2EDuration="6.194438446s" podCreationTimestamp="2026-06-02 16:38:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:38:10.192802219 +0000 UTC m=+3087.068328381" watchObservedRunningTime="2026-06-02 16:38:10.194438446 +0000 UTC m=+3087.069964607"
Jun 02 16:38:11.175713 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:11.175684    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:17.185485 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:17.185449    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:38:47.227641 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:47.227583    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 02 16:38:57.188431 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:38:57.188345    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:39:04.170735 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.170696    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:39:04.171199 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.171130    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kserve-container" containerID="cri-o://111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706" gracePeriod=30
Jun 02 16:39:04.171343 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.171202    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kube-rbac-proxy" containerID="cri-o://8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069" gracePeriod=30
Jun 02 16:39:04.251333 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251301    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:04.251806 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251785    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kube-rbac-proxy"
Jun 02 16:39:04.251891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251808    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kube-rbac-proxy"
Jun 02 16:39:04.251891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251834    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="storage-initializer"
Jun 02 16:39:04.251891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251840    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="storage-initializer"
Jun 02 16:39:04.251891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251853    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container"
Jun 02 16:39:04.251891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251859    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container"
Jun 02 16:39:04.252068 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251920    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kserve-container"
Jun 02 16:39:04.252068 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.251928    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb732b68-9580-4713-96b7-404487c7330c" containerName="kube-rbac-proxy"
Jun 02 16:39:04.256420 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.256403    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.258741 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.258715    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:39:04.258741 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.258715    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jun 02 16:39:04.264774 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.264752    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:04.360921 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.360892    2571 generic.go:358] "Generic (PLEG): container finished" podID="2b361932-3069-4ff6-b0df-311003ba670e" containerID="8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069" exitCode=2
Jun 02 16:39:04.361087 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.360963    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerDied","Data":"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"}
Jun 02 16:39:04.383339 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.383305    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.383517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.383350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.383517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.383426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.383517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.383447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2f4\" (UniqueName: \"kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.484730 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.484696    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.484908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.484738    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.484908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.484773    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.484908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.484791    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2f4\" (UniqueName: \"kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.485178 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.485140    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.485504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.485480    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.487200 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.487181    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.493383 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.493339    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2f4\" (UniqueName: \"kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4\") pod \"isvc-sklearn-runtime-predictor-5d845c599d-lz2l7\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.568502 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.568462    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:04.700934 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:04.700854    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:04.704942 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:39:04.704910    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e62772_a7bd_42f3_b1bd_781cebe3eedd.slice/crio-3cd1746c4b915d17faab94f487771cfcd2a9b0f2a00d3c344d4d1d92d376d785 WatchSource:0}: Error finding container 3cd1746c4b915d17faab94f487771cfcd2a9b0f2a00d3c344d4d1d92d376d785: Status 404 returned error can't find the container with id 3cd1746c4b915d17faab94f487771cfcd2a9b0f2a00d3c344d4d1d92d376d785
Jun 02 16:39:05.365267 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:05.365227    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerStarted","Data":"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"}
Jun 02 16:39:05.365267 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:05.365272    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerStarted","Data":"3cd1746c4b915d17faab94f487771cfcd2a9b0f2a00d3c344d4d1d92d376d785"}
Jun 02 16:39:07.180411 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:07.180353    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jun 02 16:39:10.383414 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:10.383360    2571 generic.go:358] "Generic (PLEG): container finished" podID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerID="187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b" exitCode=0
Jun 02 16:39:10.383782 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:10.383435    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerDied","Data":"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"}
Jun 02 16:39:11.388261 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.388229    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerStarted","Data":"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"}
Jun 02 16:39:11.388676 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.388270    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerStarted","Data":"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"}
Jun 02 16:39:11.388676 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.388618    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:11.388764 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.388743    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:11.390310 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.390282    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 02 16:39:11.409309 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.409252    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podStartSLOduration=7.409234651 podStartE2EDuration="7.409234651s" podCreationTimestamp="2026-06-02 16:39:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:39:11.407342786 +0000 UTC m=+3148.282868946" watchObservedRunningTime="2026-06-02 16:39:11.409234651 +0000 UTC m=+3148.284760813"
Jun 02 16:39:11.917121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:11.917090    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:39:12.051941 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.051852    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"2b361932-3069-4ff6-b0df-311003ba670e\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") "
Jun 02 16:39:12.052086 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.051984    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnqd4\" (UniqueName: \"kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4\") pod \"2b361932-3069-4ff6-b0df-311003ba670e\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") "
Jun 02 16:39:12.052086 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.052035    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location\") pod \"2b361932-3069-4ff6-b0df-311003ba670e\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") "
Jun 02 16:39:12.052174 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.052087    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls\") pod \"2b361932-3069-4ff6-b0df-311003ba670e\" (UID: \"2b361932-3069-4ff6-b0df-311003ba670e\") "
Jun 02 16:39:12.052174 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.052156    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "2b361932-3069-4ff6-b0df-311003ba670e" (UID: "2b361932-3069-4ff6-b0df-311003ba670e"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:39:12.052350 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.052332    2571 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b361932-3069-4ff6-b0df-311003ba670e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:12.052407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.052338    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b361932-3069-4ff6-b0df-311003ba670e" (UID: "2b361932-3069-4ff6-b0df-311003ba670e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:39:12.054133 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.054105    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b361932-3069-4ff6-b0df-311003ba670e" (UID: "2b361932-3069-4ff6-b0df-311003ba670e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:39:12.054275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.054179    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4" (OuterVolumeSpecName: "kube-api-access-nnqd4") pod "2b361932-3069-4ff6-b0df-311003ba670e" (UID: "2b361932-3069-4ff6-b0df-311003ba670e"). InnerVolumeSpecName "kube-api-access-nnqd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:39:12.153037 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.152989    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnqd4\" (UniqueName: \"kubernetes.io/projected/2b361932-3069-4ff6-b0df-311003ba670e-kube-api-access-nnqd4\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:12.153037 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.153021    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b361932-3069-4ff6-b0df-311003ba670e-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:12.153037 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.153032    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b361932-3069-4ff6-b0df-311003ba670e-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:12.393123 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393032    2571 generic.go:358] "Generic (PLEG): container finished" podID="2b361932-3069-4ff6-b0df-311003ba670e" containerID="111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706" exitCode=0
Jun 02 16:39:12.393620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393129    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"
Jun 02 16:39:12.393620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393128    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerDied","Data":"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"}
Jun 02 16:39:12.393620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393172    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7" event={"ID":"2b361932-3069-4ff6-b0df-311003ba670e","Type":"ContainerDied","Data":"28b505b3dbfb9a38d6f87291a44c126697b2569b87f7028beba9fb5c176e1aff"}
Jun 02 16:39:12.393620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393193    2571 scope.go:117] "RemoveContainer" containerID="8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"
Jun 02 16:39:12.393944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.393916    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 02 16:39:12.403552 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.403535    2571 scope.go:117] "RemoveContainer" containerID="111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"
Jun 02 16:39:12.413155 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.413124    2571 scope.go:117] "RemoveContainer" containerID="cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"
Jun 02 16:39:12.415067 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.415040    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:39:12.418207 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.418183    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-2wgp7"]
Jun 02 16:39:12.424041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.424022    2571 scope.go:117] "RemoveContainer" containerID="8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"
Jun 02 16:39:12.424410 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:12.424360    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069\": container with ID starting with 8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069 not found: ID does not exist" containerID="8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"
Jun 02 16:39:12.424504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.424422    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069"} err="failed to get container status \"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069\": rpc error: code = NotFound desc = could not find container \"8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069\": container with ID starting with 8f4374ec3223ad7a65835f103bf3330f664f23ed00aa717402be23d55a428069 not found: ID does not exist"
Jun 02 16:39:12.424504 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.424447    2571 scope.go:117] "RemoveContainer" containerID="111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"
Jun 02 16:39:12.424714 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:12.424697    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706\": container with ID starting with 111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706 not found: ID does not exist" containerID="111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"
Jun 02 16:39:12.424767 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.424720    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706"} err="failed to get container status \"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706\": rpc error: code = NotFound desc = could not find container \"111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706\": container with ID starting with 111150b788c638f0f8ec4ec525a1620c5aeae8881f4f9762669d877676598706 not found: ID does not exist"
Jun 02 16:39:12.424767 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.424738    2571 scope.go:117] "RemoveContainer" containerID="cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"
Jun 02 16:39:12.425012 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:12.424991    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3\": container with ID starting with cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3 not found: ID does not exist" containerID="cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"
Jun 02 16:39:12.425055 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:12.425022    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3"} err="failed to get container status \"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3\": rpc error: code = NotFound desc = could not find container \"cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3\": container with ID starting with cefa41c23d36ca2bf7bce30e8ff6c34c9cfd22ae5a8804b6f76e03cbc92a26f3 not found: ID does not exist"
Jun 02 16:39:13.684357 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:13.684316    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b361932-3069-4ff6-b0df-311003ba670e" path="/var/lib/kubelet/pods/2b361932-3069-4ff6-b0df-311003ba670e/volumes"
Jun 02 16:39:17.398164 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:17.398133    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:17.398761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:17.398733    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 02 16:39:27.399334 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:27.399299    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:41.211494 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.211460    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-5d845c599d-lz2l7_e3e62772-a7bd-42f3-b1bd-781cebe3eedd/kserve-container/0.log"
Jun 02 16:39:41.355301 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.355259    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:41.355764 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.355700    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container" containerID="cri-o://e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634" gracePeriod=30
Jun 02 16:39:41.355905 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.355746    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kube-rbac-proxy" containerID="cri-o://3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385" gracePeriod=30
Jun 02 16:39:41.447492 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447454    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:39:41.447839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447819    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kube-rbac-proxy"
Jun 02 16:39:41.447839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447835    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kube-rbac-proxy"
Jun 02 16:39:41.447839 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447847    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kserve-container"
Jun 02 16:39:41.448064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447853    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kserve-container"
Jun 02 16:39:41.448064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447866    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="storage-initializer"
Jun 02 16:39:41.448064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447873    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="storage-initializer"
Jun 02 16:39:41.448064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447930    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kube-rbac-proxy"
Jun 02 16:39:41.448064 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.447939    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b361932-3069-4ff6-b0df-311003ba670e" containerName="kserve-container"
Jun 02 16:39:41.451307 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.451289    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.453498 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.453479    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jun 02 16:39:41.453675 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.453656    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:39:41.464416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.464345    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:39:41.503282 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.503248    2571 generic.go:358] "Generic (PLEG): container finished" podID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerID="3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385" exitCode=2
Jun 02 16:39:41.503464 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.503320    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerDied","Data":"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"}
Jun 02 16:39:41.513738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.513711    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmgx\" (UniqueName: \"kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.513913 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.513777    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.513913 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.513819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.513913 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.513870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615134 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615095    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615311 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615143    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615311 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615178    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615311 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615212    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmgx\" (UniqueName: \"kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615311 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:41.615247    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 02 16:39:41.615524 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:41.615320    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls podName:144c2f3e-b15f-4fd1-adc2-2b334160dcf3 nodeName:}" failed. No retries permitted until 2026-06-02 16:39:42.115297296 +0000 UTC m=+3178.990823434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" (UID: "144c2f3e-b15f-4fd1-adc2-2b334160dcf3") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 02 16:39:41.615616 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615597    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.615886 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.615869    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:41.623152 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:41.623126    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmgx\" (UniqueName: \"kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:42.119845 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.119817    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:42.122186 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.122168    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:42.308034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.308012    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:42.322437 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.322414    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls\") pod \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") "
Jun 02 16:39:42.322549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.322463    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh2f4\" (UniqueName: \"kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4\") pod \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") "
Jun 02 16:39:42.322549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.322515    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") "
Jun 02 16:39:42.322549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.322533    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location\") pod \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\" (UID: \"e3e62772-a7bd-42f3-b1bd-781cebe3eedd\") "
Jun 02 16:39:42.322878 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.322857    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "e3e62772-a7bd-42f3-b1bd-781cebe3eedd" (UID: "e3e62772-a7bd-42f3-b1bd-781cebe3eedd"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:39:42.324439 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.324419    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3e62772-a7bd-42f3-b1bd-781cebe3eedd" (UID: "e3e62772-a7bd-42f3-b1bd-781cebe3eedd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:39:42.324498 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.324472    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4" (OuterVolumeSpecName: "kube-api-access-dh2f4") pod "e3e62772-a7bd-42f3-b1bd-781cebe3eedd" (UID: "e3e62772-a7bd-42f3-b1bd-781cebe3eedd"). InnerVolumeSpecName "kube-api-access-dh2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:39:42.349651 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.349570    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3e62772-a7bd-42f3-b1bd-781cebe3eedd" (UID: "e3e62772-a7bd-42f3-b1bd-781cebe3eedd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:39:42.361488 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.361455    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:42.423559 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.423528    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:42.423559 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.423559    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2f4\" (UniqueName: \"kubernetes.io/projected/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kube-api-access-dh2f4\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:42.423747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.423571    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:42.423747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.423581    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e62772-a7bd-42f3-b1bd-781cebe3eedd-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:39:42.491783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.491754    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:39:42.494538 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:39:42.494488    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144c2f3e_b15f_4fd1_adc2_2b334160dcf3.slice/crio-81b055baaf4830598bf8c59da92b9b7dcc4fffd7c1630c3f15c13228bdce6e9d WatchSource:0}: Error finding container 81b055baaf4830598bf8c59da92b9b7dcc4fffd7c1630c3f15c13228bdce6e9d: Status 404 returned error can't find the container with id 81b055baaf4830598bf8c59da92b9b7dcc4fffd7c1630c3f15c13228bdce6e9d
Jun 02 16:39:42.508610 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.508583    2571 generic.go:358] "Generic (PLEG): container finished" podID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerID="e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634" exitCode=0
Jun 02 16:39:42.508733 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.508672    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"
Jun 02 16:39:42.508801 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.508671    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerDied","Data":"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"}
Jun 02 16:39:42.508801 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.508778    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7" event={"ID":"e3e62772-a7bd-42f3-b1bd-781cebe3eedd","Type":"ContainerDied","Data":"3cd1746c4b915d17faab94f487771cfcd2a9b0f2a00d3c344d4d1d92d376d785"}
Jun 02 16:39:42.508801 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.508800    2571 scope.go:117] "RemoveContainer" containerID="3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"
Jun 02 16:39:42.509902 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.509878    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerStarted","Data":"81b055baaf4830598bf8c59da92b9b7dcc4fffd7c1630c3f15c13228bdce6e9d"}
Jun 02 16:39:42.521844 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.521822    2571 scope.go:117] "RemoveContainer" containerID="e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"
Jun 02 16:39:42.536402 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.536361    2571 scope.go:117] "RemoveContainer" containerID="187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"
Jun 02 16:39:42.545642 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.545613    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:42.550072 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.550052    2571 scope.go:117] "RemoveContainer" containerID="3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"
Jun 02 16:39:42.550419 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:42.550400    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385\": container with ID starting with 3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385 not found: ID does not exist" containerID="3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"
Jun 02 16:39:42.550510 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.550427    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385"} err="failed to get container status \"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385\": rpc error: code = NotFound desc = could not find container \"3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385\": container with ID starting with 3d0a43a0bb03dd97f54081f6053c3d804d83b14c31bfcf8c4c005b94e192a385 not found: ID does not exist"
Jun 02 16:39:42.550510 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.550445    2571 scope.go:117] "RemoveContainer" containerID="e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"
Jun 02 16:39:42.550759 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:42.550737    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634\": container with ID starting with e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634 not found: ID does not exist" containerID="e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"
Jun 02 16:39:42.550861 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.550768    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634"} err="failed to get container status \"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634\": rpc error: code = NotFound desc = could not find container \"e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634\": container with ID starting with e91bdd42752663e629473289c95519502ebd0cb7643161f761c1b52fc7543634 not found: ID does not exist"
Jun 02 16:39:42.550861 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.550798    2571 scope.go:117] "RemoveContainer" containerID="187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"
Jun 02 16:39:42.551119 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:39:42.551096    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b\": container with ID starting with 187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b not found: ID does not exist" containerID="187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"
Jun 02 16:39:42.551205 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.551129    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b"} err="failed to get container status \"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b\": rpc error: code = NotFound desc = could not find container \"187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b\": container with ID starting with 187b5b5b6f5a804a63005fe121dd7f698825b4dba50103b88ea3986b66d00c8b not found: ID does not exist"
Jun 02 16:39:42.551205 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:42.551170    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-5d845c599d-lz2l7"]
Jun 02 16:39:43.515814 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:43.515777    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerStarted","Data":"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"}
Jun 02 16:39:43.684944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:43.684864    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" path="/var/lib/kubelet/pods/e3e62772-a7bd-42f3-b1bd-781cebe3eedd/volumes"
Jun 02 16:39:46.528322 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:46.528287    2571 generic.go:358] "Generic (PLEG): container finished" podID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerID="5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94" exitCode=0
Jun 02 16:39:46.528322 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:46.528327    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerDied","Data":"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"}
Jun 02 16:39:47.534761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:47.534723    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerStarted","Data":"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"}
Jun 02 16:39:47.534761 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:47.534766    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerStarted","Data":"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"}
Jun 02 16:39:47.535245 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:47.535007    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:47.535245 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:47.535146    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:39:47.553203 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:47.553158    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podStartSLOduration=6.55314447 podStartE2EDuration="6.55314447s" podCreationTimestamp="2026-06-02 16:39:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:39:47.551526618 +0000 UTC m=+3184.427052778" watchObservedRunningTime="2026-06-02 16:39:47.55314447 +0000 UTC m=+3184.428670650"
Jun 02 16:39:53.543024 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:39:53.542990    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:40:23.627698 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:23.627651    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 02 16:40:33.545337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:33.545306    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:40:41.522181 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.522144    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:40:41.522755 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.522493    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container" containerID="cri-o://116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1" gracePeriod=30
Jun 02 16:40:41.522755 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.522519    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy" containerID="cri-o://b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621" gracePeriod=30
Jun 02 16:40:41.625395 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.625338    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:40:41.626029 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626000    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kube-rbac-proxy"
Jun 02 16:40:41.626029 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626029    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kube-rbac-proxy"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626065    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626076    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626092    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="storage-initializer"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626103    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="storage-initializer"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626197    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kube-rbac-proxy"
Jun 02 16:40:41.626224 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.626212    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="e3e62772-a7bd-42f3-b1bd-781cebe3eedd" containerName="kserve-container"
Jun 02 16:40:41.630287 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.630261    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.632405 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.632357    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jun 02 16:40:41.632405 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.632402    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jun 02 16:40:41.640474 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.640447    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:40:41.714706 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.714670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.714899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.714862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckwjb\" (UniqueName: \"kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.715048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.715025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.715148 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.715076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.719317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.719292    2571 generic.go:358] "Generic (PLEG): container finished" podID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerID="b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621" exitCode=2
Jun 02 16:40:41.719457 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.719350    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerDied","Data":"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"}
Jun 02 16:40:41.815621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.815516    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.815621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.815569    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.815621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.815592    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.815931 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.815648    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ckwjb\" (UniqueName: \"kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.815931 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:41.815664    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:40:41.815931 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:41.815737    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls podName:eaf88a63-7fa2-41d9-894a-b32776625454 nodeName:}" failed. No retries permitted until 2026-06-02 16:40:42.315715659 +0000 UTC m=+3239.191241799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls") pod "isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:40:41.816087 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.816006    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.816281 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.816261    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:41.825401 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:41.825352    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckwjb\" (UniqueName: \"kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:42.320025 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:42.319987    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:42.320215 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:42.320128    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:40:42.320215 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:42.320193    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls podName:eaf88a63-7fa2-41d9-894a-b32776625454 nodeName:}" failed. No retries permitted until 2026-06-02 16:40:43.320178096 +0000 UTC m=+3240.195704235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls") pod "isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jun 02 16:40:43.327840 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.327793    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:43.330351 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.330327    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:43.441969 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.441929    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:43.538820 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.538767    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jun 02 16:40:43.571706 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.571673    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:40:43.574660 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:40:43.574631    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf88a63_7fa2_41d9_894a_b32776625454.slice/crio-79293bf276b50503533bccee53779df7da232ccb24564d9143fb35c612693a2f WatchSource:0}: Error finding container 79293bf276b50503533bccee53779df7da232ccb24564d9143fb35c612693a2f: Status 404 returned error can't find the container with id 79293bf276b50503533bccee53779df7da232ccb24564d9143fb35c612693a2f
Jun 02 16:40:43.728156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.728117    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerStarted","Data":"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"}
Jun 02 16:40:43.728156 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:43.728165    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerStarted","Data":"79293bf276b50503533bccee53779df7da232ccb24564d9143fb35c612693a2f"}
Jun 02 16:40:44.586592 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:44.586548    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jun 02 16:40:47.745184 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:47.745143    2571 generic.go:358] "Generic (PLEG): container finished" podID="eaf88a63-7fa2-41d9-894a-b32776625454" containerID="86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea" exitCode=0
Jun 02 16:40:47.745621 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:47.745196    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerDied","Data":"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"}
Jun 02 16:40:48.538871 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:48.538823    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jun 02 16:40:48.750758 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:48.750723    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerStarted","Data":"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"}
Jun 02 16:40:48.750758 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:48.750765    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerStarted","Data":"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"}
Jun 02 16:40:48.751192 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:48.750968    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:48.769584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:48.769535    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podStartSLOduration=7.76952161 podStartE2EDuration="7.76952161s" podCreationTimestamp="2026-06-02 16:40:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:40:48.768222114 +0000 UTC m=+3245.643748275" watchObservedRunningTime="2026-06-02 16:40:48.76952161 +0000 UTC m=+3245.645047770"
Jun 02 16:40:49.271853 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.271830    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:40:49.372458 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372331    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") "
Jun 02 16:40:49.372458 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372418    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdmgx\" (UniqueName: \"kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx\") pod \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") "
Jun 02 16:40:49.372691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372488    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location\") pod \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") "
Jun 02 16:40:49.372691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372537    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") pod \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\" (UID: \"144c2f3e-b15f-4fd1-adc2-2b334160dcf3\") "
Jun 02 16:40:49.372822 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372723    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "144c2f3e-b15f-4fd1-adc2-2b334160dcf3" (UID: "144c2f3e-b15f-4fd1-adc2-2b334160dcf3"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:40:49.372868 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.372831    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "144c2f3e-b15f-4fd1-adc2-2b334160dcf3" (UID: "144c2f3e-b15f-4fd1-adc2-2b334160dcf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:40:49.374538 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.374515    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "144c2f3e-b15f-4fd1-adc2-2b334160dcf3" (UID: "144c2f3e-b15f-4fd1-adc2-2b334160dcf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:40:49.374637 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.374620    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx" (OuterVolumeSpecName: "kube-api-access-hdmgx") pod "144c2f3e-b15f-4fd1-adc2-2b334160dcf3" (UID: "144c2f3e-b15f-4fd1-adc2-2b334160dcf3"). InnerVolumeSpecName "kube-api-access-hdmgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:40:49.473548 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.473509    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:40:49.473548 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.473540    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:40:49.473548 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.473553    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:40:49.473790 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.473563    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmgx\" (UniqueName: \"kubernetes.io/projected/144c2f3e-b15f-4fd1-adc2-2b334160dcf3-kube-api-access-hdmgx\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:40:49.756339 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756303    2571 generic.go:358] "Generic (PLEG): container finished" podID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerID="116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1" exitCode=0
Jun 02 16:40:49.756795 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756408    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"
Jun 02 16:40:49.756795 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756405    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerDied","Data":"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"}
Jun 02 16:40:49.756795 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756520    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz" event={"ID":"144c2f3e-b15f-4fd1-adc2-2b334160dcf3","Type":"ContainerDied","Data":"81b055baaf4830598bf8c59da92b9b7dcc4fffd7c1630c3f15c13228bdce6e9d"}
Jun 02 16:40:49.756795 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756537    2571 scope.go:117] "RemoveContainer" containerID="b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"
Jun 02 16:40:49.756992 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.756949    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:49.758487 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.758438    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:40:49.765059 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.765043    2571 scope.go:117] "RemoveContainer" containerID="116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"
Jun 02 16:40:49.773108 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.773083    2571 scope.go:117] "RemoveContainer" containerID="5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"
Jun 02 16:40:49.773992 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.773971    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:40:49.778434 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.778412    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-qhcgz"]
Jun 02 16:40:49.781245 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.781231    2571 scope.go:117] "RemoveContainer" containerID="b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"
Jun 02 16:40:49.781512 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:49.781492    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621\": container with ID starting with b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621 not found: ID does not exist" containerID="b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"
Jun 02 16:40:49.781568 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.781521    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621"} err="failed to get container status \"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621\": rpc error: code = NotFound desc = could not find container \"b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621\": container with ID starting with b361e8279be12fe2bcc36536d7e3c42af6a0935f97eb63beb1e73d800c611621 not found: ID does not exist"
Jun 02 16:40:49.781568 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.781540    2571 scope.go:117] "RemoveContainer" containerID="116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"
Jun 02 16:40:49.781768 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:49.781754    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1\": container with ID starting with 116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1 not found: ID does not exist" containerID="116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"
Jun 02 16:40:49.781809 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.781772    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1"} err="failed to get container status \"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1\": rpc error: code = NotFound desc = could not find container \"116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1\": container with ID starting with 116dc57f92fac925f25b7bf86e4b2312c4fa2963d7976366d83f12de2ca020e1 not found: ID does not exist"
Jun 02 16:40:49.781809 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.781786    2571 scope.go:117] "RemoveContainer" containerID="5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"
Jun 02 16:40:49.782001 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:40:49.781985    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94\": container with ID starting with 5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94 not found: ID does not exist" containerID="5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"
Jun 02 16:40:49.782049 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:49.782006    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94"} err="failed to get container status \"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94\": rpc error: code = NotFound desc = could not find container \"5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94\": container with ID starting with 5bf45e5b6b3fe13dbce20bd041189f5e0f9e996a055aae3690444c9e826d8e94 not found: ID does not exist"
Jun 02 16:40:50.761552 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:50.761512    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:40:51.684501 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:51.684467    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" path="/var/lib/kubelet/pods/144c2f3e-b15f-4fd1-adc2-2b334160dcf3/volumes"
Jun 02 16:40:55.766898 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:55.766870    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:40:55.767544 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:40:55.767513    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:05.767490 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:05.767449    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:15.768121 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:15.768079    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:25.767548 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:25.767508    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:35.768383 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:35.768328    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:43.858540 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:43.858448    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:41:43.864283 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:43.864260    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:41:45.767778 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:45.767735    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:41:55.768615 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:41:55.768530    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:42:01.858843 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.858810    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:42:01.859447 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.859089    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" containerID="cri-o://bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552" gracePeriod=30
Jun 02 16:42:01.859447 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.859150    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kube-rbac-proxy" containerID="cri-o://3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5" gracePeriod=30
Jun 02 16:42:01.954988 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.954950    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:42:01.955335 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955323    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="storage-initializer"
Jun 02 16:42:01.955407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955337    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="storage-initializer"
Jun 02 16:42:01.955407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955347    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container"
Jun 02 16:42:01.955407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955354    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container"
Jun 02 16:42:01.955407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955379    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy"
Jun 02 16:42:01.955407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955385    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy"
Jun 02 16:42:01.955575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955443    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kube-rbac-proxy"
Jun 02 16:42:01.955575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.955453    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="144c2f3e-b15f-4fd1-adc2-2b334160dcf3" containerName="kserve-container"
Jun 02 16:42:01.959989 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.959970    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:01.962290 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.962266    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jun 02 16:42:01.962417 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.962266    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jun 02 16:42:01.967391 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.967347    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:42:01.998995 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.998961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:01.999143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.999013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7mz\" (UniqueName: \"kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:01.999143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.999081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:01.999235 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:01.999186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.003252 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.003225    2571 generic.go:358] "Generic (PLEG): container finished" podID="eaf88a63-7fa2-41d9-894a-b32776625454" containerID="3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5" exitCode=2
Jun 02 16:42:02.003398 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.003269    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerDied","Data":"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"}
Jun 02 16:42:02.100181 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100145    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.100404 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100189    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.100404 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100220    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-th7mz\" (UniqueName: \"kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.100404 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100240    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.100404 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:42:02.100317    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jun 02 16:42:02.100404 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:42:02.100407    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls podName:2987002d-4989-4b0f-975d-1e8928be07a6 nodeName:}" failed. No retries permitted until 2026-06-02 16:42:02.600362994 +0000 UTC m=+3319.475889132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" (UID: "2987002d-4989-4b0f-975d-1e8928be07a6") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jun 02 16:42:02.100650 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100600    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.100961 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.100942    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.108998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.108912    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7mz\" (UniqueName: \"kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.605605 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.605567    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.607991 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.607960    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:02.872507 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:02.872412    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:03.004564 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:03.004530    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:42:03.011926 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:03.011908    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:42:04.011331 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:04.011289    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerStarted","Data":"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"}
Jun 02 16:42:04.011331 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:04.011333    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerStarted","Data":"41e18a9cead33f14ff2f1d91e41498dad9dd15828282e28ee5387999853aedce"}
Jun 02 16:42:05.762231 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:05.762192    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jun 02 16:42:05.767917 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:05.767889    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jun 02 16:42:06.307093 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.307068    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:42:06.438410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438295    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location\") pod \"eaf88a63-7fa2-41d9-894a-b32776625454\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") "
Jun 02 16:42:06.438410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438343    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") pod \"eaf88a63-7fa2-41d9-894a-b32776625454\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") "
Jun 02 16:42:06.438410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438397    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckwjb\" (UniqueName: \"kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb\") pod \"eaf88a63-7fa2-41d9-894a-b32776625454\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") "
Jun 02 16:42:06.438651 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438487    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"eaf88a63-7fa2-41d9-894a-b32776625454\" (UID: \"eaf88a63-7fa2-41d9-894a-b32776625454\") "
Jun 02 16:42:06.438691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438658    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eaf88a63-7fa2-41d9-894a-b32776625454" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:42:06.438924 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.438900    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "eaf88a63-7fa2-41d9-894a-b32776625454" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:42:06.440624 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.440596    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eaf88a63-7fa2-41d9-894a-b32776625454" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:42:06.440727 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.440639    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb" (OuterVolumeSpecName: "kube-api-access-ckwjb") pod "eaf88a63-7fa2-41d9-894a-b32776625454" (UID: "eaf88a63-7fa2-41d9-894a-b32776625454"). InnerVolumeSpecName "kube-api-access-ckwjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:42:06.539180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.539143    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaf88a63-7fa2-41d9-894a-b32776625454-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:42:06.539180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.539173    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaf88a63-7fa2-41d9-894a-b32776625454-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:42:06.539180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.539184    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaf88a63-7fa2-41d9-894a-b32776625454-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:42:06.539456 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:06.539196    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckwjb\" (UniqueName: \"kubernetes.io/projected/eaf88a63-7fa2-41d9-894a-b32776625454-kube-api-access-ckwjb\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:42:07.022272 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.022182    2571 generic.go:358] "Generic (PLEG): container finished" podID="2987002d-4989-4b0f-975d-1e8928be07a6" containerID="c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b" exitCode=0
Jun 02 16:42:07.022272 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.022257    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerDied","Data":"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"}
Jun 02 16:42:07.024139 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.024117    2571 generic.go:358] "Generic (PLEG): container finished" podID="eaf88a63-7fa2-41d9-894a-b32776625454" containerID="bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552" exitCode=0
Jun 02 16:42:07.024251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.024194    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"
Jun 02 16:42:07.024251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.024193    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerDied","Data":"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"}
Jun 02 16:42:07.024251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.024226    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm" event={"ID":"eaf88a63-7fa2-41d9-894a-b32776625454","Type":"ContainerDied","Data":"79293bf276b50503533bccee53779df7da232ccb24564d9143fb35c612693a2f"}
Jun 02 16:42:07.024251 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.024242    2571 scope.go:117] "RemoveContainer" containerID="3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"
Jun 02 16:42:07.036531 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.036510    2571 scope.go:117] "RemoveContainer" containerID="bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"
Jun 02 16:42:07.045305 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.045285    2571 scope.go:117] "RemoveContainer" containerID="86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"
Jun 02 16:42:07.054824 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.054795    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:42:07.057421 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.057384    2571 scope.go:117] "RemoveContainer" containerID="3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"
Jun 02 16:42:07.057818 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:42:07.057794    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5\": container with ID starting with 3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5 not found: ID does not exist" containerID="3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"
Jun 02 16:42:07.057900 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.057827    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5"} err="failed to get container status \"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5\": rpc error: code = NotFound desc = could not find container \"3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5\": container with ID starting with 3a3a2951a23395732ba2d91a72ba26e0a2306d8aeb51cf8c68a1477df80dd8e5 not found: ID does not exist"
Jun 02 16:42:07.057900 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.057845    2571 scope.go:117] "RemoveContainer" containerID="bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"
Jun 02 16:42:07.058165 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:42:07.058141    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552\": container with ID starting with bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552 not found: ID does not exist" containerID="bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"
Jun 02 16:42:07.058280 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.058168    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552"} err="failed to get container status \"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552\": rpc error: code = NotFound desc = could not find container \"bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552\": container with ID starting with bd8549d24e89f0a6a6557918cebb38e9fb11fcdec925f4660da7e1cd709ca552 not found: ID does not exist"
Jun 02 16:42:07.058280 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.058199    2571 scope.go:117] "RemoveContainer" containerID="86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"
Jun 02 16:42:07.058578 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:42:07.058552    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea\": container with ID starting with 86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea not found: ID does not exist" containerID="86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"
Jun 02 16:42:07.058678 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.058588    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea"} err="failed to get container status \"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea\": rpc error: code = NotFound desc = could not find container \"86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea\": container with ID starting with 86d6af2d1158a2cb8611388e706bcf30486a15ab7a1bfd7aa0d30006d01188ea not found: ID does not exist"
Jun 02 16:42:07.061167 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.061145    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-57fbb95bb6-ntrfm"]
Jun 02 16:42:07.684117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:07.684085    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" path="/var/lib/kubelet/pods/eaf88a63-7fa2-41d9-894a-b32776625454/volumes"
Jun 02 16:42:08.032762 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:08.032725    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerStarted","Data":"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"}
Jun 02 16:42:08.033253 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:08.032773    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerStarted","Data":"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"}
Jun 02 16:42:08.033253 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:08.033112    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:08.051421 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:08.051356    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podStartSLOduration=7.05134267 podStartE2EDuration="7.05134267s" podCreationTimestamp="2026-06-02 16:42:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:42:08.05028427 +0000 UTC m=+3324.925810432" watchObservedRunningTime="2026-06-02 16:42:08.05134267 +0000 UTC m=+3324.926868830"
Jun 02 16:42:09.037649 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:09.037615    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:09.038958 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:09.038931    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:10.040771 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:10.040730    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:15.046521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:15.046488    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:42:15.047068 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:15.047041    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:25.047194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:25.047154    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:35.047060 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:35.047015    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:45.047942 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:45.047900    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:42:55.048042 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:42:55.047998    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:43:05.047034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:05.046990    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:43:15.048542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:15.048508    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:43:22.044438 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:22.044401    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:43:22.044899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:22.044735    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" containerID="cri-o://9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45" gracePeriod=30
Jun 02 16:43:22.044899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:22.044781    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kube-rbac-proxy" containerID="cri-o://d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106" gracePeriod=30
Jun 02 16:43:22.284019 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:22.283982    2571 generic.go:358] "Generic (PLEG): container finished" podID="2987002d-4989-4b0f-975d-1e8928be07a6" containerID="d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106" exitCode=2
Jun 02 16:43:22.284188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:22.284060    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerDied","Data":"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"}
Jun 02 16:43:25.041806 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:25.041756    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jun 02 16:43:25.047186 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:25.047158    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 02 16:43:26.495871 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.495848    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:43:26.546493 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546462    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7mz\" (UniqueName: \"kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz\") pod \"2987002d-4989-4b0f-975d-1e8928be07a6\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") "
Jun 02 16:43:26.546652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546560    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location\") pod \"2987002d-4989-4b0f-975d-1e8928be07a6\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") "
Jun 02 16:43:26.546652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546599    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"2987002d-4989-4b0f-975d-1e8928be07a6\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") "
Jun 02 16:43:26.546652 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546633    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") pod \"2987002d-4989-4b0f-975d-1e8928be07a6\" (UID: \"2987002d-4989-4b0f-975d-1e8928be07a6\") "
Jun 02 16:43:26.546957 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546934    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2987002d-4989-4b0f-975d-1e8928be07a6" (UID: "2987002d-4989-4b0f-975d-1e8928be07a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:43:26.547027 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.546984    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "2987002d-4989-4b0f-975d-1e8928be07a6" (UID: "2987002d-4989-4b0f-975d-1e8928be07a6"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:43:26.548578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.548543    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz" (OuterVolumeSpecName: "kube-api-access-th7mz") pod "2987002d-4989-4b0f-975d-1e8928be07a6" (UID: "2987002d-4989-4b0f-975d-1e8928be07a6"). InnerVolumeSpecName "kube-api-access-th7mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:43:26.548684 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.548592    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2987002d-4989-4b0f-975d-1e8928be07a6" (UID: "2987002d-4989-4b0f-975d-1e8928be07a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:43:26.647747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.647658    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2987002d-4989-4b0f-975d-1e8928be07a6-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:43:26.647747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.647689    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2987002d-4989-4b0f-975d-1e8928be07a6-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:43:26.647747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.647706    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2987002d-4989-4b0f-975d-1e8928be07a6-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:43:26.647747 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:26.647721    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7mz\" (UniqueName: \"kubernetes.io/projected/2987002d-4989-4b0f-975d-1e8928be07a6-kube-api-access-th7mz\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:43:27.300612 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.300576    2571 generic.go:358] "Generic (PLEG): container finished" podID="2987002d-4989-4b0f-975d-1e8928be07a6" containerID="9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45" exitCode=0
Jun 02 16:43:27.300793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.300660    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerDied","Data":"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"}
Jun 02 16:43:27.300793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.300701    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj" event={"ID":"2987002d-4989-4b0f-975d-1e8928be07a6","Type":"ContainerDied","Data":"41e18a9cead33f14ff2f1d91e41498dad9dd15828282e28ee5387999853aedce"}
Jun 02 16:43:27.300793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.300716    2571 scope.go:117] "RemoveContainer" containerID="d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"
Jun 02 16:43:27.300793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.300669    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"
Jun 02 16:43:27.310165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.310150    2571 scope.go:117] "RemoveContainer" containerID="9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"
Jun 02 16:43:27.317556 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.317539    2571 scope.go:117] "RemoveContainer" containerID="c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"
Jun 02 16:43:27.323143 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.323113    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:43:27.326350 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326333    2571 scope.go:117] "RemoveContainer" containerID="d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"
Jun 02 16:43:27.326654 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:43:27.326632    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106\": container with ID starting with d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106 not found: ID does not exist" containerID="d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"
Jun 02 16:43:27.326776 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326658    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106"} err="failed to get container status \"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106\": rpc error: code = NotFound desc = could not find container \"d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106\": container with ID starting with d6d463bb54fefcaba3fd3be5929229481be9667b1b79fa44c4e8d5175da14106 not found: ID does not exist"
Jun 02 16:43:27.326776 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326677    2571 scope.go:117] "RemoveContainer" containerID="9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"
Jun 02 16:43:27.326893 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326865    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8585bf8d59-wjhtj"]
Jun 02 16:43:27.326943 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:43:27.326916    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45\": container with ID starting with 9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45 not found: ID does not exist" containerID="9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"
Jun 02 16:43:27.326981 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326942    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45"} err="failed to get container status \"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45\": rpc error: code = NotFound desc = could not find container \"9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45\": container with ID starting with 9e4891db3068b3556206c283145ff46741671df38f3a91654dfdeab54c708f45 not found: ID does not exist"
Jun 02 16:43:27.326981 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.326956    2571 scope.go:117] "RemoveContainer" containerID="c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"
Jun 02 16:43:27.327171 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:43:27.327156    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b\": container with ID starting with c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b not found: ID does not exist" containerID="c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"
Jun 02 16:43:27.327210 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.327173    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b"} err="failed to get container status \"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b\": rpc error: code = NotFound desc = could not find container \"c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b\": container with ID starting with c27cefb698780e8a0bd03dc99d83dde2953c3b4a3bad07d24797db521770688b not found: ID does not exist"
Jun 02 16:43:27.685213 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:43:27.685130    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" path="/var/lib/kubelet/pods/2987002d-4989-4b0f-975d-1e8928be07a6/volumes"
Jun 02 16:46:43.882353 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:46:43.882318    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:46:43.889846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:46:43.889824    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:48:38.685822 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.685782    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686296    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kube-rbac-proxy"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686317    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kube-rbac-proxy"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686336    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="storage-initializer"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686345    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="storage-initializer"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686361    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686389    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686403    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686412    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686424    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="storage-initializer"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686432    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="storage-initializer"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686450    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kube-rbac-proxy"
Jun 02 16:48:38.686483 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686458    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kube-rbac-proxy"
Jun 02 16:48:38.687189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686541    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kserve-container"
Jun 02 16:48:38.687189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686554    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="eaf88a63-7fa2-41d9-894a-b32776625454" containerName="kube-rbac-proxy"
Jun 02 16:48:38.687189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686565    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kube-rbac-proxy"
Jun 02 16:48:38.687189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.686581    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="2987002d-4989-4b0f-975d-1e8928be07a6" containerName="kserve-container"
Jun 02 16:48:38.690054 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.690034    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.693517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.693496    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jun 02 16:48:38.693644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.693628    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:48:38.694010 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.693995    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 02 16:48:38.694164 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.694147    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:48:38.694453 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.694437    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:48:38.703112 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.703091    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:48:38.836477 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.836438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.836667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.836508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxq7\" (UniqueName: \"kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.836667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.836545    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.836667 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.836580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937193 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937105    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937193 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937156    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937428 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937195    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937428 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937235    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxq7\" (UniqueName: \"kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937428 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:48:38.937346    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jun 02 16:48:38.937561 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:48:38.937460    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls podName:391fd7c1-64fb-4f7b-88ee-a0f212d3234b nodeName:}" failed. No retries permitted until 2026-06-02 16:48:39.437437497 +0000 UTC m=+3716.312963655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" (UID: "391fd7c1-64fb-4f7b-88ee-a0f212d3234b") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jun 02 16:48:38.937614 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937597    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.937831 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.937813    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:38.957122 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:38.957093    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxq7\" (UniqueName: \"kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:39.442632 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:39.442585    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:39.445053 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:39.445032    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:39.600195 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:39.600157    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:39.734811 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:39.734781    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:48:39.737612 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:48:39.737579    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391fd7c1_64fb_4f7b_88ee_a0f212d3234b.slice/crio-771e108f19f78178d8811f7a3075e02184a0d6c8a597d9e0e0702436f91129c9 WatchSource:0}: Error finding container 771e108f19f78178d8811f7a3075e02184a0d6c8a597d9e0e0702436f91129c9: Status 404 returned error can't find the container with id 771e108f19f78178d8811f7a3075e02184a0d6c8a597d9e0e0702436f91129c9
Jun 02 16:48:39.739645 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:39.739623    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:48:40.340557 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:40.340517    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerStarted","Data":"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"}
Jun 02 16:48:40.340557 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:40.340555    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerStarted","Data":"771e108f19f78178d8811f7a3075e02184a0d6c8a597d9e0e0702436f91129c9"}
Jun 02 16:48:44.352989 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:44.352950    2571 generic.go:358] "Generic (PLEG): container finished" podID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerID="73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3" exitCode=0
Jun 02 16:48:44.353455 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:44.352995    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerDied","Data":"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"}
Jun 02 16:48:45.358432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:45.358397    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerStarted","Data":"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"}
Jun 02 16:48:45.358432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:45.358439    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerStarted","Data":"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"}
Jun 02 16:48:45.358856 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:45.358635    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:45.378125 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:45.378072    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" podStartSLOduration=7.378057213 podStartE2EDuration="7.378057213s" podCreationTimestamp="2026-06-02 16:48:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:48:45.377875831 +0000 UTC m=+3722.253401992" watchObservedRunningTime="2026-06-02 16:48:45.378057213 +0000 UTC m=+3722.253583374"
Jun 02 16:48:46.361805 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:46.361773    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:48:52.371059 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:48:52.371029    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:49:22.374582 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:22.374550    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:49:28.841416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.841379    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:49:28.847310 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.847284    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:28.849765 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.849743    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jun 02 16:49:28.849924 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.849901    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 02 16:49:28.856403 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.856350    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:49:28.856790 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.856761    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kserve-container" containerID="cri-o://01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381" gracePeriod=30
Jun 02 16:49:28.856908 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.856818    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kube-rbac-proxy" containerID="cri-o://4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921" gracePeriod=30
Jun 02 16:49:28.860802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.860774    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:49:28.973743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.973706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:28.973895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.973753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:28.973895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.973807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98ds\" (UniqueName: \"kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:28.973895 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:28.973840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.074690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.074644    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.074690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.074695    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.074956 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.074732    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s98ds\" (UniqueName: \"kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.074956 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.074759    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.075146 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.075121    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.075430 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.075412    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.077141 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.077118    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.082183 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.082163    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98ds\" (UniqueName: \"kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-n68pq\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.164321 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.164243    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:29.295101 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.295076    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:49:29.297389 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:49:29.297346    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4703859a_9693_472a_9fe7_d345800b45af.slice/crio-c0877d7961909df59d814ba896858fde4db53d2b13cbb68fd34a6038bbe3223f WatchSource:0}: Error finding container c0877d7961909df59d814ba896858fde4db53d2b13cbb68fd34a6038bbe3223f: Status 404 returned error can't find the container with id c0877d7961909df59d814ba896858fde4db53d2b13cbb68fd34a6038bbe3223f
Jun 02 16:49:29.510427 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.510385    2571 generic.go:358] "Generic (PLEG): container finished" podID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerID="4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921" exitCode=2
Jun 02 16:49:29.510427 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.510406    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerDied","Data":"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"}
Jun 02 16:49:29.511744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.511714    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerStarted","Data":"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"}
Jun 02 16:49:29.511744 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:29.511743    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerStarted","Data":"c0877d7961909df59d814ba896858fde4db53d2b13cbb68fd34a6038bbe3223f"}
Jun 02 16:49:32.366690 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:32.366644    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jun 02 16:49:32.372276 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:32.372248    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jun 02 16:49:33.526362 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:33.526327    2571 generic.go:358] "Generic (PLEG): container finished" podID="4703859a-9693-472a-9fe7-d345800b45af" containerID="195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18" exitCode=0
Jun 02 16:49:33.526831 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:33.526402    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerDied","Data":"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"}
Jun 02 16:49:34.533812 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:34.533764    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerStarted","Data":"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"}
Jun 02 16:49:34.534217 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:34.533827    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerStarted","Data":"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"}
Jun 02 16:49:34.534217 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:34.534116    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:34.534297 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:34.534221    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:49:34.554523 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:34.554469    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" podStartSLOduration=6.554456479 podStartE2EDuration="6.554456479s" podCreationTimestamp="2026-06-02 16:49:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:49:34.552533266 +0000 UTC m=+3771.428059427" watchObservedRunningTime="2026-06-02 16:49:34.554456479 +0000 UTC m=+3771.429982639"
Jun 02 16:49:35.714397 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.714360    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:49:35.834731 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.834632    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") pod \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") "
Jun 02 16:49:35.834731 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.834691    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gxq7\" (UniqueName: \"kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7\") pod \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") "
Jun 02 16:49:35.834731 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.834715    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") "
Jun 02 16:49:35.835020 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.834759    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location\") pod \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\" (UID: \"391fd7c1-64fb-4f7b-88ee-a0f212d3234b\") "
Jun 02 16:49:35.835166 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.835138    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "391fd7c1-64fb-4f7b-88ee-a0f212d3234b" (UID: "391fd7c1-64fb-4f7b-88ee-a0f212d3234b"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:49:35.835239 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.835182    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "391fd7c1-64fb-4f7b-88ee-a0f212d3234b" (UID: "391fd7c1-64fb-4f7b-88ee-a0f212d3234b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:49:35.836813 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.836789    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7" (OuterVolumeSpecName: "kube-api-access-6gxq7") pod "391fd7c1-64fb-4f7b-88ee-a0f212d3234b" (UID: "391fd7c1-64fb-4f7b-88ee-a0f212d3234b"). InnerVolumeSpecName "kube-api-access-6gxq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:49:35.836889 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.836840    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "391fd7c1-64fb-4f7b-88ee-a0f212d3234b" (UID: "391fd7c1-64fb-4f7b-88ee-a0f212d3234b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:49:35.935416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.935380    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:49:35.935416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.935411    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxq7\" (UniqueName: \"kubernetes.io/projected/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kube-api-access-6gxq7\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:49:35.935416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.935423    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:49:35.935647 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:35.935433    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/391fd7c1-64fb-4f7b-88ee-a0f212d3234b-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:49:36.542516 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.542485    2571 generic.go:358] "Generic (PLEG): container finished" podID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerID="01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381" exitCode=0
Jun 02 16:49:36.542708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.542535    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerDied","Data":"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"}
Jun 02 16:49:36.542708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.542558    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf" event={"ID":"391fd7c1-64fb-4f7b-88ee-a0f212d3234b","Type":"ContainerDied","Data":"771e108f19f78178d8811f7a3075e02184a0d6c8a597d9e0e0702436f91129c9"}
Jun 02 16:49:36.542708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.542565    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"
Jun 02 16:49:36.542708 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.542572    2571 scope.go:117] "RemoveContainer" containerID="4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"
Jun 02 16:49:36.551288 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.551271    2571 scope.go:117] "RemoveContainer" containerID="01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"
Jun 02 16:49:36.558997 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.558978    2571 scope.go:117] "RemoveContainer" containerID="73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"
Jun 02 16:49:36.564952 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.564931    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:49:36.566960 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.566944    2571 scope.go:117] "RemoveContainer" containerID="4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"
Jun 02 16:49:36.567231 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:49:36.567210    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921\": container with ID starting with 4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921 not found: ID does not exist" containerID="4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"
Jun 02 16:49:36.567325 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.567240    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921"} err="failed to get container status \"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921\": rpc error: code = NotFound desc = could not find container \"4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921\": container with ID starting with 4a00519a2f947df4f05778d5f87213fdfdcc063abfc7580fe7cff4f427068921 not found: ID does not exist"
Jun 02 16:49:36.567325 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.567258    2571 scope.go:117] "RemoveContainer" containerID="01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"
Jun 02 16:49:36.567558 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:49:36.567536    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381\": container with ID starting with 01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381 not found: ID does not exist" containerID="01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"
Jun 02 16:49:36.567617 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.567565    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381"} err="failed to get container status \"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381\": rpc error: code = NotFound desc = could not find container \"01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381\": container with ID starting with 01b17074b37ff403c0e11e19c9188225c5b6629541afa2377a189723a55f1381 not found: ID does not exist"
Jun 02 16:49:36.567617 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.567586    2571 scope.go:117] "RemoveContainer" containerID="73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"
Jun 02 16:49:36.567825 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:49:36.567800    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3\": container with ID starting with 73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3 not found: ID does not exist" containerID="73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"
Jun 02 16:49:36.567886 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.567833    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3"} err="failed to get container status \"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3\": rpc error: code = NotFound desc = could not find container \"73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3\": container with ID starting with 73a59c7bb25fa3010a4f82ffa6b16af8904872df3606dcfbcd9856c6b32941b3 not found: ID does not exist"
Jun 02 16:49:36.571270 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:36.571249    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-9p5vf"]
Jun 02 16:49:37.684489 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:37.684457    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" path="/var/lib/kubelet/pods/391fd7c1-64fb-4f7b-88ee-a0f212d3234b/volumes"
Jun 02 16:49:40.543443 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:49:40.543413    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:50:10.547846 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:10.547815    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:50:18.943644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:18.943606    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:50:18.944069 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:18.943960    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kserve-container" containerID="cri-o://a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d" gracePeriod=30
Jun 02 16:50:18.944069 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:18.944004    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy" containerID="cri-o://f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d" gracePeriod=30
Jun 02 16:50:19.685010 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:19.684977    2571 generic.go:358] "Generic (PLEG): container finished" podID="4703859a-9693-472a-9fe7-d345800b45af" containerID="f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d" exitCode=2
Jun 02 16:50:19.685190 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:19.685049    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerDied","Data":"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"}
Jun 02 16:50:20.538150 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:20.538104    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jun 02 16:50:25.591378 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.591344    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:50:25.659423 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659387    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s98ds\" (UniqueName: \"kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds\") pod \"4703859a-9693-472a-9fe7-d345800b45af\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") "
Jun 02 16:50:25.659584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659474    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location\") pod \"4703859a-9693-472a-9fe7-d345800b45af\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") "
Jun 02 16:50:25.659584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659505    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls\") pod \"4703859a-9693-472a-9fe7-d345800b45af\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") "
Jun 02 16:50:25.659584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659523    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"4703859a-9693-472a-9fe7-d345800b45af\" (UID: \"4703859a-9693-472a-9fe7-d345800b45af\") "
Jun 02 16:50:25.659869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659848    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4703859a-9693-472a-9fe7-d345800b45af" (UID: "4703859a-9693-472a-9fe7-d345800b45af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.659950 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.659908    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "4703859a-9693-472a-9fe7-d345800b45af" (UID: "4703859a-9693-472a-9fe7-d345800b45af"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:50:25.661530 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.661508    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4703859a-9693-472a-9fe7-d345800b45af" (UID: "4703859a-9693-472a-9fe7-d345800b45af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:50:25.661611 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.661514    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds" (OuterVolumeSpecName: "kube-api-access-s98ds") pod "4703859a-9693-472a-9fe7-d345800b45af" (UID: "4703859a-9693-472a-9fe7-d345800b45af"). InnerVolumeSpecName "kube-api-access-s98ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:50:25.706252 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.706218    2571 generic.go:358] "Generic (PLEG): container finished" podID="4703859a-9693-472a-9fe7-d345800b45af" containerID="a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d" exitCode=0
Jun 02 16:50:25.706387 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.706300    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerDied","Data":"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"}
Jun 02 16:50:25.706387 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.706345    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" event={"ID":"4703859a-9693-472a-9fe7-d345800b45af","Type":"ContainerDied","Data":"c0877d7961909df59d814ba896858fde4db53d2b13cbb68fd34a6038bbe3223f"}
Jun 02 16:50:25.706387 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.706361    2571 scope.go:117] "RemoveContainer" containerID="f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"
Jun 02 16:50:25.706566 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.706310    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"
Jun 02 16:50:25.714381 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.714343    2571 scope.go:117] "RemoveContainer" containerID="a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"
Jun 02 16:50:25.724230 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.722553    2571 scope.go:117] "RemoveContainer" containerID="195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"
Jun 02 16:50:25.726713 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.726680    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:50:25.730120 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.730099    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq"]
Jun 02 16:50:25.732861 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.732843    2571 scope.go:117] "RemoveContainer" containerID="f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"
Jun 02 16:50:25.733125 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:50:25.733099    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d\": container with ID starting with f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d not found: ID does not exist" containerID="f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"
Jun 02 16:50:25.733181 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.733132    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d"} err="failed to get container status \"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d\": rpc error: code = NotFound desc = could not find container \"f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d\": container with ID starting with f3331007dcd3aad0a234d6478259cc164834697723cad31030f420271e6e094d not found: ID does not exist"
Jun 02 16:50:25.733181 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.733150    2571 scope.go:117] "RemoveContainer" containerID="a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"
Jun 02 16:50:25.733437 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:50:25.733417    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d\": container with ID starting with a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d not found: ID does not exist" containerID="a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"
Jun 02 16:50:25.733489 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.733445    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d"} err="failed to get container status \"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d\": rpc error: code = NotFound desc = could not find container \"a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d\": container with ID starting with a443ec4705ee831c2e71673250d6fb9b4e98e514876aa0c0622e70a84149d54d not found: ID does not exist"
Jun 02 16:50:25.733489 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.733461    2571 scope.go:117] "RemoveContainer" containerID="195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"
Jun 02 16:50:25.733711 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:50:25.733693    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18\": container with ID starting with 195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18 not found: ID does not exist" containerID="195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"
Jun 02 16:50:25.733766 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.733720    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18"} err="failed to get container status \"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18\": rpc error: code = NotFound desc = could not find container \"195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18\": container with ID starting with 195c8e8d17ae7f336b4bb5f0c89484d5d1b12b0e87ed1db1d898fd792b0a8b18 not found: ID does not exist"
Jun 02 16:50:25.760262 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.760230    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4703859a-9693-472a-9fe7-d345800b45af-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.760262 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.760259    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4703859a-9693-472a-9fe7-d345800b45af-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.760467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.760271    2571 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4703859a-9693-472a-9fe7-d345800b45af-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.760467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:25.760280    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98ds\" (UniqueName: \"kubernetes.io/projected/4703859a-9693-472a-9fe7-d345800b45af-kube-api-access-s98ds\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:50:26.538754 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:26.538708    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-n68pq" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": context deadline exceeded"
Jun 02 16:50:27.688789 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:50:27.688749    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4703859a-9693-472a-9fe7-d345800b45af" path="/var/lib/kubelet/pods/4703859a-9693-472a-9fe7-d345800b45af/volumes"
Jun 02 16:51:39.208694 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.208659    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209009    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="storage-initializer"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209019    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="storage-initializer"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209028    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209034    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209047    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="storage-initializer"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209053    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="storage-initializer"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209059    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kube-rbac-proxy"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209064    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kube-rbac-proxy"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209070    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209075    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209092    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209097    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209149    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209158    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kserve-container"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209165    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="4703859a-9693-472a-9fe7-d345800b45af" containerName="kube-rbac-proxy"
Jun 02 16:51:39.209188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.209176    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="391fd7c1-64fb-4f7b-88ee-a0f212d3234b" containerName="kube-rbac-proxy"
Jun 02 16:51:39.212376 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.212340    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.214953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.214930    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 02 16:51:39.215089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.215003    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:51:39.215089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.215028    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jun 02 16:51:39.215089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.215003    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:51:39.215089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.215058    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:51:39.224319 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.224296    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:51:39.295071 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.295031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnz27\" (UniqueName: \"kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.295266 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.295090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.295266 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.295193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.295410 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.295319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.396289 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.396249    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz27\" (UniqueName: \"kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.396502 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.396302    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.396502 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.396340    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.396502 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.396430    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.396910 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.396882    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.397088 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.397069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.398734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.398710    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.404442 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.404419    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz27\" (UniqueName: \"kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.523216 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.523165    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:39.652552 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.652525    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:51:39.654954 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:51:39.654920    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e16d8dd_8258_488a_bbe3_cc1660dbf5a8.slice/crio-02850ed7ff6311095e9eff0465353333b361cda6354a54d508e7a928993d5a13 WatchSource:0}: Error finding container 02850ed7ff6311095e9eff0465353333b361cda6354a54d508e7a928993d5a13: Status 404 returned error can't find the container with id 02850ed7ff6311095e9eff0465353333b361cda6354a54d508e7a928993d5a13
Jun 02 16:51:39.953714 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.953623    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerStarted","Data":"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"}
Jun 02 16:51:39.953714 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:39.953666    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerStarted","Data":"02850ed7ff6311095e9eff0465353333b361cda6354a54d508e7a928993d5a13"}
Jun 02 16:51:43.910267 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:43.910184    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:51:43.917886 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:43.917866    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:51:43.970325 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:43.970288    2571 generic.go:358] "Generic (PLEG): container finished" podID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerID="5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463" exitCode=0
Jun 02 16:51:43.970517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:43.970384    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerDied","Data":"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"}
Jun 02 16:51:44.975227 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:44.975192    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerStarted","Data":"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"}
Jun 02 16:51:44.975802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:44.975238    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerStarted","Data":"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"}
Jun 02 16:51:44.975802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:44.975477    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:44.975802 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:44.975502    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:51:44.994521 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:44.994465    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podStartSLOduration=5.994448863 podStartE2EDuration="5.994448863s" podCreationTimestamp="2026-06-02 16:51:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:51:44.992055922 +0000 UTC m=+3901.867582074" watchObservedRunningTime="2026-06-02 16:51:44.994448863 +0000 UTC m=+3901.869975026"
Jun 02 16:51:50.984111 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:51:50.984078    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:52:21.027799 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:21.027752    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 02 16:52:30.986647 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:30.986612    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:52:39.293487 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:39.293449    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:52:39.294455 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:39.294419    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container" containerID="cri-o://e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0" gracePeriod=30
Jun 02 16:52:39.294999 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:39.294855    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy" containerID="cri-o://cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5" gracePeriod=30
Jun 02 16:52:40.160947 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:40.160911    2571 generic.go:358] "Generic (PLEG): container finished" podID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerID="cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5" exitCode=2
Jun 02 16:52:40.160947 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:40.160930    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerDied","Data":"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"}
Jun 02 16:52:40.978732 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:40.978681    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jun 02 16:52:42.025610 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:42.025556    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jun 02 16:52:45.978899 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:45.978853    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jun 02 16:52:46.936909 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:46.936887    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:52:47.074122 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074032    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location\") pod \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") "
Jun 02 16:52:47.074122 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074114    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls\") pod \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") "
Jun 02 16:52:47.074664 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074165    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") "
Jun 02 16:52:47.074664 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074195    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnz27\" (UniqueName: \"kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27\") pod \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\" (UID: \"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8\") "
Jun 02 16:52:47.074664 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074447    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" (UID: "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:52:47.074664 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.074559    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" (UID: "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:52:47.076204 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.076172    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" (UID: "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:52:47.076312 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.076202    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27" (OuterVolumeSpecName: "kube-api-access-qnz27") pod "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" (UID: "0e16d8dd-8258-488a-bbe3-cc1660dbf5a8"). InnerVolumeSpecName "kube-api-access-qnz27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:52:47.175292 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.175259    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:52:47.175292 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.175291    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnz27\" (UniqueName: \"kubernetes.io/projected/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kube-api-access-qnz27\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:52:47.175538 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.175304    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:52:47.175538 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.175313    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:52:47.191579 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.191543    2571 generic.go:358] "Generic (PLEG): container finished" podID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerID="e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0" exitCode=0
Jun 02 16:52:47.191738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.191634    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"
Jun 02 16:52:47.191738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.191628    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerDied","Data":"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"}
Jun 02 16:52:47.191813 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.191743    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d" event={"ID":"0e16d8dd-8258-488a-bbe3-cc1660dbf5a8","Type":"ContainerDied","Data":"02850ed7ff6311095e9eff0465353333b361cda6354a54d508e7a928993d5a13"}
Jun 02 16:52:47.191813 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.191759    2571 scope.go:117] "RemoveContainer" containerID="cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"
Jun 02 16:52:47.200037 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.200016    2571 scope.go:117] "RemoveContainer" containerID="e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"
Jun 02 16:52:47.207135 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.207115    2571 scope.go:117] "RemoveContainer" containerID="5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"
Jun 02 16:52:47.212494 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.212471    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:52:47.215175 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215149    2571 scope.go:117] "RemoveContainer" containerID="cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"
Jun 02 16:52:47.215462 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:52:47.215432    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5\": container with ID starting with cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5 not found: ID does not exist" containerID="cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"
Jun 02 16:52:47.215549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215474    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5"} err="failed to get container status \"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5\": rpc error: code = NotFound desc = could not find container \"cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5\": container with ID starting with cad0a6918131bcfd95ddb8bb7975fa4c4f5f2c49daaa73fc6c4e02538e3c37f5 not found: ID does not exist"
Jun 02 16:52:47.215549 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215499    2571 scope.go:117] "RemoveContainer" containerID="e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"
Jun 02 16:52:47.215646 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215581    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-f875d"]
Jun 02 16:52:47.215754 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:52:47.215735    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0\": container with ID starting with e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0 not found: ID does not exist" containerID="e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"
Jun 02 16:52:47.215793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215762    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0"} err="failed to get container status \"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0\": rpc error: code = NotFound desc = could not find container \"e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0\": container with ID starting with e3334c171acdfbc3f207f473c01b979bed8ebc1da08b8a5d9a394a18db30c5f0 not found: ID does not exist"
Jun 02 16:52:47.215793 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.215779    2571 scope.go:117] "RemoveContainer" containerID="5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"
Jun 02 16:52:47.216016 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:52:47.216001    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463\": container with ID starting with 5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463 not found: ID does not exist" containerID="5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"
Jun 02 16:52:47.216054 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.216022    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463"} err="failed to get container status \"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463\": rpc error: code = NotFound desc = could not find container \"5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463\": container with ID starting with 5a5ba339b725a0b541df5a0f2e898ab367182ed6f230d7dcbffd8cb2b46f3463 not found: ID does not exist"
Jun 02 16:52:47.684104 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:52:47.684070    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" path="/var/lib/kubelet/pods/0e16d8dd-8258-488a-bbe3-cc1660dbf5a8/volumes"
Jun 02 16:53:59.552074 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.551996    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552345    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552355    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552382    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="storage-initializer"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552388    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="storage-initializer"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552405    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552411    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552460    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kube-rbac-proxy"
Jun 02 16:53:59.552518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.552470    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e16d8dd-8258-488a-bbe3-cc1660dbf5a8" containerName="kserve-container"
Jun 02 16:53:59.555749 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.555726    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.558053 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558033    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jun 02 16:53:59.558489 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558472    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:53:59.558578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558492    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jun 02 16:53:59.558578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558513    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ztrcr\""
Jun 02 16:53:59.558578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558564    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jun 02 16:53:59.558739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.558514    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:53:59.566227 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.566201    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:53:59.683089 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.683054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.683268 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.683126    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.683268 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.683162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhh9\" (UniqueName: \"kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.683268 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.683252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.784626 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.784586    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.784833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.784641    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhh9\" (UniqueName: \"kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.784833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.784687    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.784833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.784715    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.785154 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.785129    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.785341 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.785321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.787151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.787133    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.793164 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.793139    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhh9\" (UniqueName: \"kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9\") pod \"isvc-sklearn-s3-predictor-555695658c-mt6qd\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.867892 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.867791    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:53:59.995722 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:53:59.995685    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:53:59.999425 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:53:59.999397    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78105e79_bc65_4407_916e_e683a47afe26.slice/crio-4b4f940f05984ab6e8cf458c6648f60f557211907e6991f5f54fea32c461cf9b WatchSource:0}: Error finding container 4b4f940f05984ab6e8cf458c6648f60f557211907e6991f5f54fea32c461cf9b: Status 404 returned error can't find the container with id 4b4f940f05984ab6e8cf458c6648f60f557211907e6991f5f54fea32c461cf9b
Jun 02 16:54:00.001432 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:00.001417    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:54:00.435715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:00.435670    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerStarted","Data":"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"}
Jun 02 16:54:00.435715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:00.435709    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerStarted","Data":"4b4f940f05984ab6e8cf458c6648f60f557211907e6991f5f54fea32c461cf9b"}
Jun 02 16:54:01.441479 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:01.441378    2571 generic.go:358] "Generic (PLEG): container finished" podID="78105e79-bc65-4407-916e-e683a47afe26" containerID="5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded" exitCode=0
Jun 02 16:54:01.441479 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:01.441429    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerDied","Data":"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"}
Jun 02 16:54:02.446165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.446129    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerStarted","Data":"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"}
Jun 02 16:54:02.446165 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.446164    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerStarted","Data":"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"}
Jun 02 16:54:02.446620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.446339    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:54:02.446620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.446469    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:54:02.447707 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.447682    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:02.466565 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:02.466520    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podStartSLOduration=3.466507748 podStartE2EDuration="3.466507748s" podCreationTimestamp="2026-06-02 16:53:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:54:02.464014817 +0000 UTC m=+4039.339540978" watchObservedRunningTime="2026-06-02 16:54:02.466507748 +0000 UTC m=+4039.342033908"
Jun 02 16:54:03.449965 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:03.449922    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:08.454274 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:08.454240    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:54:08.454760 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:08.454709    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:18.454966 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:18.454927    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:28.454853 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:28.454809    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:38.455033 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:38.454991    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:48.454709 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:48.454658    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:54:58.455317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:54:58.455268    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 02 16:55:08.455474 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:08.455441    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:55:09.668745 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.668712    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:55:09.669184 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.669088    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container" containerID="cri-o://df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7" gracePeriod=30
Jun 02 16:55:09.669184 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.669120    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kube-rbac-proxy" containerID="cri-o://33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d" gracePeriod=30
Jun 02 16:55:09.795511 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.795478    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:55:09.799081 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.799061    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:09.801041 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.801024    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jun 02 16:55:09.801235 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.801217    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jun 02 16:55:09.801418 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.801403    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 02 16:55:09.810408 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.810358    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:55:09.902280 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.902243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:09.902280 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.902290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:09.902520 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.902332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:09.902520 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.902392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:09.902520 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:09.902441    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxlv\" (UniqueName: \"kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.003709 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.003672    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.003709 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.003714    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.003973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.003742    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.003973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.003765    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.003973 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.003902    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxlv\" (UniqueName: \"kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.004140 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:55:10.003919    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jun 02 16:55:10.004140 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:55:10.004053    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls podName:924029f5-5040-4004-9d47-c3dc2cbb3c71 nodeName:}" failed. No retries permitted until 2026-06-02 16:55:10.50403102 +0000 UTC m=+4107.379557188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jun 02 16:55:10.004264 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.004232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.004418 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.004401    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.004464 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.004431    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.014349 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.014319    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxlv\" (UniqueName: \"kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.508569 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.508529    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.510960 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.510932    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.675517 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.675486    2571 generic.go:358] "Generic (PLEG): container finished" podID="78105e79-bc65-4407-916e-e683a47afe26" containerID="33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d" exitCode=2
Jun 02 16:55:10.675887 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.675564    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerDied","Data":"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"}
Jun 02 16:55:10.709900 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.709866    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:10.842762 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:10.842717    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:55:10.846343 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:55:10.846315    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924029f5_5040_4004_9d47_c3dc2cbb3c71.slice/crio-e8bbafb1e4c3c647fa27986b64c3b46689221ce918a625e4c1bb060c8007e8c7 WatchSource:0}: Error finding container e8bbafb1e4c3c647fa27986b64c3b46689221ce918a625e4c1bb060c8007e8c7: Status 404 returned error can't find the container with id e8bbafb1e4c3c647fa27986b64c3b46689221ce918a625e4c1bb060c8007e8c7
Jun 02 16:55:11.685021 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:11.684980    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerStarted","Data":"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"}
Jun 02 16:55:11.685448 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:11.685029    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerStarted","Data":"e8bbafb1e4c3c647fa27986b64c3b46689221ce918a625e4c1bb060c8007e8c7"}
Jun 02 16:55:12.684094 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:12.684056    2571 generic.go:358] "Generic (PLEG): container finished" podID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerID="de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1" exitCode=0
Jun 02 16:55:12.684282 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:12.684141    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerDied","Data":"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"}
Jun 02 16:55:13.450783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:13.450738    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jun 02 16:55:13.688401 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:13.688352    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerStarted","Data":"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"}
Jun 02 16:55:13.688401 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:13.688405    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerStarted","Data":"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"}
Jun 02 16:55:13.688669 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:13.688519    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:13.706433 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:13.706312    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podStartSLOduration=4.706285427 podStartE2EDuration="4.706285427s" podCreationTimestamp="2026-06-02 16:55:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:55:13.705213405 +0000 UTC m=+4110.580739565" watchObservedRunningTime="2026-06-02 16:55:13.706285427 +0000 UTC m=+4110.581811589"
Jun 02 16:55:14.322972 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.322950    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:55:14.341759 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.341714    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls\") pod \"78105e79-bc65-4407-916e-e683a47afe26\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") "
Jun 02 16:55:14.341759 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.341762    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location\") pod \"78105e79-bc65-4407-916e-e683a47afe26\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") "
Jun 02 16:55:14.342216 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.342177    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78105e79-bc65-4407-916e-e683a47afe26" (UID: "78105e79-bc65-4407-916e-e683a47afe26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:55:14.344303 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.344276    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78105e79-bc65-4407-916e-e683a47afe26" (UID: "78105e79-bc65-4407-916e-e683a47afe26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:55:14.443045 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.442963    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"78105e79-bc65-4407-916e-e683a47afe26\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") "
Jun 02 16:55:14.443045 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.443002    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqhh9\" (UniqueName: \"kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9\") pod \"78105e79-bc65-4407-916e-e683a47afe26\" (UID: \"78105e79-bc65-4407-916e-e683a47afe26\") "
Jun 02 16:55:14.443230 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.443141    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78105e79-bc65-4407-916e-e683a47afe26-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:55:14.443230 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.443154    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78105e79-bc65-4407-916e-e683a47afe26-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:55:14.443320 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.443296    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "78105e79-bc65-4407-916e-e683a47afe26" (UID: "78105e79-bc65-4407-916e-e683a47afe26"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:55:14.445012 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.444984    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9" (OuterVolumeSpecName: "kube-api-access-rqhh9") pod "78105e79-bc65-4407-916e-e683a47afe26" (UID: "78105e79-bc65-4407-916e-e683a47afe26"). InnerVolumeSpecName "kube-api-access-rqhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:55:14.544467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.544424    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78105e79-bc65-4407-916e-e683a47afe26-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:55:14.544467 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.544459    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqhh9\" (UniqueName: \"kubernetes.io/projected/78105e79-bc65-4407-916e-e683a47afe26-kube-api-access-rqhh9\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:55:14.693725 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.693631    2571 generic.go:358] "Generic (PLEG): container finished" podID="78105e79-bc65-4407-916e-e683a47afe26" containerID="df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7" exitCode=0
Jun 02 16:55:14.693869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.693718    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerDied","Data":"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"}
Jun 02 16:55:14.693869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.693746    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"
Jun 02 16:55:14.693869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.693762    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd" event={"ID":"78105e79-bc65-4407-916e-e683a47afe26","Type":"ContainerDied","Data":"4b4f940f05984ab6e8cf458c6648f60f557211907e6991f5f54fea32c461cf9b"}
Jun 02 16:55:14.693869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.693790    2571 scope.go:117] "RemoveContainer" containerID="33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"
Jun 02 16:55:14.694306 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.694285    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:14.695715 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.695688    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:55:14.702907 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.702886    2571 scope.go:117] "RemoveContainer" containerID="df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"
Jun 02 16:55:14.710179 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.710160    2571 scope.go:117] "RemoveContainer" containerID="5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"
Jun 02 16:55:14.715461 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.715439    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:55:14.718317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.718295    2571 scope.go:117] "RemoveContainer" containerID="33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"
Jun 02 16:55:14.718623 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:55:14.718604    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d\": container with ID starting with 33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d not found: ID does not exist" containerID="33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"
Jun 02 16:55:14.718723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.718631    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d"} err="failed to get container status \"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d\": rpc error: code = NotFound desc = could not find container \"33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d\": container with ID starting with 33c7b710fc8dedbb03ec1026c293d5a92058b14ed51a43f8de7d9ae5dab61e1d not found: ID does not exist"
Jun 02 16:55:14.718723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.718651    2571 scope.go:117] "RemoveContainer" containerID="df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"
Jun 02 16:55:14.718911 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:55:14.718876    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7\": container with ID starting with df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7 not found: ID does not exist" containerID="df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"
Jun 02 16:55:14.718911 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.718894    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7"} err="failed to get container status \"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7\": rpc error: code = NotFound desc = could not find container \"df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7\": container with ID starting with df51517e78eca2f3007d26e65807027e16f302ba1ef7e335b1cadd57dcc842c7 not found: ID does not exist"
Jun 02 16:55:14.718911 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.718906    2571 scope.go:117] "RemoveContainer" containerID="5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"
Jun 02 16:55:14.719119 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:55:14.719102    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded\": container with ID starting with 5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded not found: ID does not exist" containerID="5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"
Jun 02 16:55:14.719181 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.719122    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded"} err="failed to get container status \"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded\": rpc error: code = NotFound desc = could not find container \"5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded\": container with ID starting with 5104e384e91b88b39b233baf6190a6728fe7d0ff9a9801339d014f2204483ded not found: ID does not exist"
Jun 02 16:55:14.719388 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:14.719345    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-555695658c-mt6qd"]
Jun 02 16:55:15.684273 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:15.684241    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78105e79-bc65-4407-916e-e683a47afe26" path="/var/lib/kubelet/pods/78105e79-bc65-4407-916e-e683a47afe26/volumes"
Jun 02 16:55:15.698250 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:15.698209    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:55:20.703112 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:20.703081    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:55:20.703691 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:20.703664    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:55:30.704322 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:30.704278    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:55:40.704092 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:40.704053    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:55:50.704018 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:55:50.703974    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:56:00.704636 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:00.704588    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:56:10.703843 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:10.703804    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:56:20.704356 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:20.704316    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:56:29.842879 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:29.842842    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:56:29.843348 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:29.843185    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" containerID="cri-o://aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b" gracePeriod=30
Jun 02 16:56:29.843348 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:29.843218    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kube-rbac-proxy" containerID="cri-o://a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331" gracePeriod=30
Jun 02 16:56:29.965331 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:29.965300    2571 generic.go:358] "Generic (PLEG): container finished" podID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerID="a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331" exitCode=2
Jun 02 16:56:29.965503 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:29.965385    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerDied","Data":"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"}
Jun 02 16:56:30.699098 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.699051    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jun 02 16:56:30.704414 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.704361    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jun 02 16:56:30.952894 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.952808    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953178    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container"
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953191    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container"
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="storage-initializer"
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953213    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="storage-initializer"
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953227    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kube-rbac-proxy"
Jun 02 16:56:30.953275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953233    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kube-rbac-proxy"
Jun 02 16:56:30.953564 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953288    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kserve-container"
Jun 02 16:56:30.953564 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.953297    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="78105e79-bc65-4407-916e-e683a47afe26" containerName="kube-rbac-proxy"
Jun 02 16:56:30.956648 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.956626    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:30.958797 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.958771    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jun 02 16:56:30.958935 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.958811    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jun 02 16:56:30.966102 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:30.966081    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:31.093066 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.093018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.093066 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.093065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.093287 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.093143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.093287 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.093188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtlk\" (UniqueName: \"kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.193863 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.193831    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtlk\" (UniqueName: \"kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.194036 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.193885    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.194036 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.193907    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.194036 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.193953    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.194412 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.194395    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.194639 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.194622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.196419 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.196398    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.202188 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.202162    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtlk\" (UniqueName: \"kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.268230 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.268191    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:31.406002 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.405941    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:31.408724 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:56:31.408693    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dec210_ea89_453f_bee0_2ea59645f499.slice/crio-30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174 WatchSource:0}: Error finding container 30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174: Status 404 returned error can't find the container with id 30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174
Jun 02 16:56:31.974153 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.974117    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerStarted","Data":"db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b"}
Jun 02 16:56:31.974153 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:31.974155    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerStarted","Data":"30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174"}
Jun 02 16:56:33.981927 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:33.981899    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/0.log"
Jun 02 16:56:33.982263 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:33.981938    2571 generic.go:358] "Generic (PLEG): container finished" podID="c2dec210-ea89-453f-bee0-2ea59645f499" containerID="db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b" exitCode=1
Jun 02 16:56:33.982263 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:33.982021    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerDied","Data":"db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b"}
Jun 02 16:56:34.203932 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.203908    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:56:34.320220 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320184    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert\") pod \"924029f5-5040-4004-9d47-c3dc2cbb3c71\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") "
Jun 02 16:56:34.320446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320252    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location\") pod \"924029f5-5040-4004-9d47-c3dc2cbb3c71\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") "
Jun 02 16:56:34.320446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320302    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") pod \"924029f5-5040-4004-9d47-c3dc2cbb3c71\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") "
Jun 02 16:56:34.320446 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320398    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"924029f5-5040-4004-9d47-c3dc2cbb3c71\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") "
Jun 02 16:56:34.320614 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320446    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqxlv\" (UniqueName: \"kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv\") pod \"924029f5-5040-4004-9d47-c3dc2cbb3c71\" (UID: \"924029f5-5040-4004-9d47-c3dc2cbb3c71\") "
Jun 02 16:56:34.320614 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320586    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "924029f5-5040-4004-9d47-c3dc2cbb3c71" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:56:34.320723 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320645    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "924029f5-5040-4004-9d47-c3dc2cbb3c71" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:56:34.320796 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320729    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-cabundle-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:34.320859 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320842    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/924029f5-5040-4004-9d47-c3dc2cbb3c71-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:34.320966 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.320937    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "924029f5-5040-4004-9d47-c3dc2cbb3c71" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:56:34.322543 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.322520    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "924029f5-5040-4004-9d47-c3dc2cbb3c71" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:56:34.322647 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.322584    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv" (OuterVolumeSpecName: "kube-api-access-lqxlv") pod "924029f5-5040-4004-9d47-c3dc2cbb3c71" (UID: "924029f5-5040-4004-9d47-c3dc2cbb3c71"). InnerVolumeSpecName "kube-api-access-lqxlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:56:34.421291 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.421252    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/924029f5-5040-4004-9d47-c3dc2cbb3c71-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:34.421291 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.421282    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxlv\" (UniqueName: \"kubernetes.io/projected/924029f5-5040-4004-9d47-c3dc2cbb3c71-kube-api-access-lqxlv\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:34.421291 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.421294    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/924029f5-5040-4004-9d47-c3dc2cbb3c71-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:34.987117 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.987088    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/0.log"
Jun 02 16:56:34.987578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.987188    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerStarted","Data":"8d9f0a914484a6a197577ff74951a9e0012d8441b50507988ad9f935c351ae71"}
Jun 02 16:56:34.988888 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.988865    2571 generic.go:358] "Generic (PLEG): container finished" podID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerID="aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b" exitCode=0
Jun 02 16:56:34.989005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.988912    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerDied","Data":"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"}
Jun 02 16:56:34.989005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.988930    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf" event={"ID":"924029f5-5040-4004-9d47-c3dc2cbb3c71","Type":"ContainerDied","Data":"e8bbafb1e4c3c647fa27986b64c3b46689221ce918a625e4c1bb060c8007e8c7"}
Jun 02 16:56:34.989005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.988945    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"
Jun 02 16:56:34.989005 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.988961    2571 scope.go:117] "RemoveContainer" containerID="a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"
Jun 02 16:56:34.997400 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:34.997348    2571 scope.go:117] "RemoveContainer" containerID="aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"
Jun 02 16:56:35.005326 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.005308    2571 scope.go:117] "RemoveContainer" containerID="de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"
Jun 02 16:56:35.013562 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.013536    2571 scope.go:117] "RemoveContainer" containerID="a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"
Jun 02 16:56:35.013821 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:35.013800    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331\": container with ID starting with a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331 not found: ID does not exist" containerID="a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"
Jun 02 16:56:35.013872 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.013834    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331"} err="failed to get container status \"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331\": rpc error: code = NotFound desc = could not find container \"a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331\": container with ID starting with a82d8a8898e538ee4755f67c1a1181a96d407c2718f172152564c46e48d0a331 not found: ID does not exist"
Jun 02 16:56:35.013872 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.013854    2571 scope.go:117] "RemoveContainer" containerID="aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"
Jun 02 16:56:35.014120 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:35.014096    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b\": container with ID starting with aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b not found: ID does not exist" containerID="aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"
Jun 02 16:56:35.014174 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.014126    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b"} err="failed to get container status \"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b\": rpc error: code = NotFound desc = could not find container \"aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b\": container with ID starting with aafb23fee16ae98792203a3e1f3553dc4b775b57cd42614f1767fbe10e70e06b not found: ID does not exist"
Jun 02 16:56:35.014174 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.014143    2571 scope.go:117] "RemoveContainer" containerID="de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"
Jun 02 16:56:35.014356 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:35.014342    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1\": container with ID starting with de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1 not found: ID does not exist" containerID="de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"
Jun 02 16:56:35.014420 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.014360    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1"} err="failed to get container status \"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1\": rpc error: code = NotFound desc = could not find container \"de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1\": container with ID starting with de9e2dd538e6d00d557e89d8fafed9efae516bbe7cf46a4962a1b7d9a290daf1 not found: ID does not exist"
Jun 02 16:56:35.018958 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.018934    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:56:35.024836 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.024813    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-56bc7f457f-lz4mf"]
Jun 02 16:56:35.683922 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:35.683889    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" path="/var/lib/kubelet/pods/924029f5-5040-4004-9d47-c3dc2cbb3c71/volumes"
Jun 02 16:56:40.006982 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.006951    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/1.log"
Jun 02 16:56:40.007449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.007312    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/0.log"
Jun 02 16:56:40.007449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.007342    2571 generic.go:358] "Generic (PLEG): container finished" podID="c2dec210-ea89-453f-bee0-2ea59645f499" containerID="8d9f0a914484a6a197577ff74951a9e0012d8441b50507988ad9f935c351ae71" exitCode=1
Jun 02 16:56:40.007449 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.007419    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerDied","Data":"8d9f0a914484a6a197577ff74951a9e0012d8441b50507988ad9f935c351ae71"}
Jun 02 16:56:40.007617 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.007462    2571 scope.go:117] "RemoveContainer" containerID="db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b"
Jun 02 16:56:40.007872 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.007853    2571 scope.go:117] "RemoveContainer" containerID="db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b"
Jun 02 16:56:40.018061 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:40.018023    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_kserve-ci-e2e-test_c2dec210-ea89-453f-bee0-2ea59645f499_0 in pod sandbox 30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174 from index: no such id: 'db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b'" containerID="db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b"
Jun 02 16:56:40.018122 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:40.018089    2571 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_kserve-ci-e2e-test_c2dec210-ea89-453f-bee0-2ea59645f499_0 in pod sandbox 30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174 from index: no such id: 'db36b7be39252e4195e8ec98fdb36eab75fc5bc7a5dd1db9f23f6c4f8d9e692b'; Skipping pod \"isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_kserve-ci-e2e-test(c2dec210-ea89-453f-bee0-2ea59645f499)\"" logger="UnhandledError"
Jun 02 16:56:40.019446 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:56:40.019425    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_kserve-ci-e2e-test(c2dec210-ea89-453f-bee0-2ea59645f499)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" podUID="c2dec210-ea89-453f-bee0-2ea59645f499"
Jun 02 16:56:40.896670 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:40.896632    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:41.012144 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.012119    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/1.log"
Jun 02 16:56:41.144900 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.144876    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/1.log"
Jun 02 16:56:41.145086 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.144937    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:41.283833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.283793    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls\") pod \"c2dec210-ea89-453f-bee0-2ea59645f499\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") "
Jun 02 16:56:41.283833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.283833    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wtlk\" (UniqueName: \"kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk\") pod \"c2dec210-ea89-453f-bee0-2ea59645f499\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") "
Jun 02 16:56:41.284080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.283968    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"c2dec210-ea89-453f-bee0-2ea59645f499\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") "
Jun 02 16:56:41.284080 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.284016    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location\") pod \"c2dec210-ea89-453f-bee0-2ea59645f499\" (UID: \"c2dec210-ea89-453f-bee0-2ea59645f499\") "
Jun 02 16:56:41.284320 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.284290    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2dec210-ea89-453f-bee0-2ea59645f499" (UID: "c2dec210-ea89-453f-bee0-2ea59645f499"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:56:41.284456 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.284340    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "c2dec210-ea89-453f-bee0-2ea59645f499" (UID: "c2dec210-ea89-453f-bee0-2ea59645f499"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:56:41.286478 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.286460    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk" (OuterVolumeSpecName: "kube-api-access-9wtlk") pod "c2dec210-ea89-453f-bee0-2ea59645f499" (UID: "c2dec210-ea89-453f-bee0-2ea59645f499"). InnerVolumeSpecName "kube-api-access-9wtlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:56:41.286557 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.286482    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2dec210-ea89-453f-bee0-2ea59645f499" (UID: "c2dec210-ea89-453f-bee0-2ea59645f499"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:56:41.385119 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.385087    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2dec210-ea89-453f-bee0-2ea59645f499-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:41.385119 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.385117    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2dec210-ea89-453f-bee0-2ea59645f499-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:41.385119 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.385128    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtlk\" (UniqueName: \"kubernetes.io/projected/c2dec210-ea89-453f-bee0-2ea59645f499-kube-api-access-9wtlk\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:41.385395 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.385138    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2dec210-ea89-453f-bee0-2ea59645f499-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:56:41.998620 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998584    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:56:41.998948 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998935    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container"
Jun 02 16:56:41.998998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998950    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container"
Jun 02 16:56:41.998998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998965    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="storage-initializer"
Jun 02 16:56:41.998998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998971    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="storage-initializer"
Jun 02 16:56:41.998998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998978    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kube-rbac-proxy"
Jun 02 16:56:41.998998 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.998983    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kube-rbac-proxy"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999006    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999011    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999018    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999023    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999073    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kserve-container"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999083    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:41.999151 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999091    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="924029f5-5040-4004-9d47-c3dc2cbb3c71" containerName="kube-rbac-proxy"
Jun 02 16:56:41.999354 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:41.999190    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" containerName="storage-initializer"
Jun 02 16:56:42.003551 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.003533    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.009565 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.009521    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 02 16:56:42.009738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.009593    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jun 02 16:56:42.009738 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.009528    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jun 02 16:56:42.012894 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.012863    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:56:42.017059 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.017041    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4_c2dec210-ea89-453f-bee0-2ea59645f499/storage-initializer/1.log"
Jun 02 16:56:42.017180 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.017158    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4" event={"ID":"c2dec210-ea89-453f-bee0-2ea59645f499","Type":"ContainerDied","Data":"30997f8339804a1555f5931d6cd105b7d065f962cabcc5d37f9b5fb0e545f174"}
Jun 02 16:56:42.017231 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.017200    2571 scope.go:117] "RemoveContainer" containerID="8d9f0a914484a6a197577ff74951a9e0012d8441b50507988ad9f935c351ae71"
Jun 02 16:56:42.017268 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.017168    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"
Jun 02 16:56:42.054097 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.054063    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:42.059515 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.059482    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-848c66f57c-z9gr4"]
Jun 02 16:56:42.092773 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.092732    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.092944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.092788    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.092944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.092870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrgl\" (UniqueName: \"kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.092944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.092902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.092944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.092933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193414 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193346    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrgl\" (UniqueName: \"kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193426    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193457    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193505    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193575 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193553    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.193974 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.193934    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.194185 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.194166    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.194234 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.194178    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.196039 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.196022    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.202110 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.202087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrgl\" (UniqueName: \"kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.315183 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.315094    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:42.446974 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:42.446947    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:56:42.449249 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:56:42.449219    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366d182_55cb_4431_b07f_49446e79a20e.slice/crio-2484da3ec6772cc08a3cb035278351a17a2c8a7611a6dcb81a26c5631bac17f1 WatchSource:0}: Error finding container 2484da3ec6772cc08a3cb035278351a17a2c8a7611a6dcb81a26c5631bac17f1: Status 404 returned error can't find the container with id 2484da3ec6772cc08a3cb035278351a17a2c8a7611a6dcb81a26c5631bac17f1
Jun 02 16:56:43.022443 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:43.022406    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerStarted","Data":"130a666da7a577b0501a1ce43ec8661051b3bbb08cba6620d70dc3306007d235"}
Jun 02 16:56:43.022443 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:43.022443    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerStarted","Data":"2484da3ec6772cc08a3cb035278351a17a2c8a7611a6dcb81a26c5631bac17f1"}
Jun 02 16:56:43.684229 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:43.684194    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2dec210-ea89-453f-bee0-2ea59645f499" path="/var/lib/kubelet/pods/c2dec210-ea89-453f-bee0-2ea59645f499/volumes"
Jun 02 16:56:43.935763 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:43.935685    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:56:43.942959 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:43.942937    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 16:56:44.028034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:44.027999    2571 generic.go:358] "Generic (PLEG): container finished" podID="7366d182-55cb-4431-b07f-49446e79a20e" containerID="130a666da7a577b0501a1ce43ec8661051b3bbb08cba6620d70dc3306007d235" exitCode=0
Jun 02 16:56:44.028411 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:44.028055    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerDied","Data":"130a666da7a577b0501a1ce43ec8661051b3bbb08cba6620d70dc3306007d235"}
Jun 02 16:56:45.034029 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:45.033992    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerStarted","Data":"2daef734dbf6a61f8895f2a029026cd030aac446f8b460794869f01fd546eef7"}
Jun 02 16:56:45.034459 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:45.034037    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerStarted","Data":"d6fc07d15a489b0d472c770c7161ea022cd933ab6f3faefbdb501d234571fdac"}
Jun 02 16:56:45.034459 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:45.034166    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:45.054653 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:45.054606    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podStartSLOduration=4.05459018 podStartE2EDuration="4.05459018s" podCreationTimestamp="2026-06-02 16:56:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:56:45.053442654 +0000 UTC m=+4201.928968816" watchObservedRunningTime="2026-06-02 16:56:45.05459018 +0000 UTC m=+4201.930116342"
Jun 02 16:56:46.037034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:46.036997    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:46.038409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:46.038355    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:56:47.040326 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:47.040285    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:56:52.044317 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:52.044240    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:56:52.044843 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:56:52.044814    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:02.045385 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:02.045328    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:12.045307 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:12.045265    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:22.044993 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:22.044951    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:32.044843 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:32.044801    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:42.044891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:42.044849    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 02 16:57:52.045971 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:57:52.045939    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:58:02.003066 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.003031    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:58:02.003560 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.003452    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container" containerID="cri-o://d6fc07d15a489b0d472c770c7161ea022cd933ab6f3faefbdb501d234571fdac" gracePeriod=30
Jun 02 16:58:02.003560 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.003497    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kube-rbac-proxy" containerID="cri-o://2daef734dbf6a61f8895f2a029026cd030aac446f8b460794869f01fd546eef7" gracePeriod=30
Jun 02 16:58:02.040867 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.040827    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jun 02 16:58:02.306033 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.305941    2571 generic.go:358] "Generic (PLEG): container finished" podID="7366d182-55cb-4431-b07f-49446e79a20e" containerID="2daef734dbf6a61f8895f2a029026cd030aac446f8b460794869f01fd546eef7" exitCode=2
Jun 02 16:58:02.306033 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:02.306016    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerDied","Data":"2daef734dbf6a61f8895f2a029026cd030aac446f8b460794869f01fd546eef7"}
Jun 02 16:58:03.092338 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.092306    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:03.097416 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.097354    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.101223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.101090    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jun 02 16:58:03.101693 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.101329    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jun 02 16:58:03.103522 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.103499    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:03.188043 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.188001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.188231 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.188053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.188231 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.188121    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.188231 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.188154    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxpk\" (UniqueName: \"kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289103    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289140 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289144    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289418 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289177    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289418 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289200    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxpk\" (UniqueName: \"kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289418 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:58:03.289337    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jun 02 16:58:03.289545 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:58:03.289437    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls podName:5b935eda-8248-4dc2-88d8-655806c228af nodeName:}" failed. No retries permitted until 2026-06-02 16:58:03.78941704 +0000 UTC m=+4280.664943197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" (UID: "5b935eda-8248-4dc2-88d8-655806c228af") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jun 02 16:58:03.289687 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289670    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.289845 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.289827    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.297211 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.297181    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxpk\" (UniqueName: \"kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.793409 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.793348    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:03.795891 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:03.795862    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:04.010912 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:04.010874    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:04.140854 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:04.140827    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:04.143918 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:58:04.143892    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b935eda_8248_4dc2_88d8_655806c228af.slice/crio-c7ae6ac127044facef668185d87f8013b0a99a102fe65616cd3bf74c72c970c7 WatchSource:0}: Error finding container c7ae6ac127044facef668185d87f8013b0a99a102fe65616cd3bf74c72c970c7: Status 404 returned error can't find the container with id c7ae6ac127044facef668185d87f8013b0a99a102fe65616cd3bf74c72c970c7
Jun 02 16:58:04.314584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:04.314493    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerStarted","Data":"bd4c6b3bf4f6c3210bda6dacf74f81838db1619463b2bdafcfca3aa13d549d56"}
Jun 02 16:58:04.314584 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:04.314534    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerStarted","Data":"c7ae6ac127044facef668185d87f8013b0a99a102fe65616cd3bf74c72c970c7"}
Jun 02 16:58:06.323783 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.323748    2571 generic.go:358] "Generic (PLEG): container finished" podID="7366d182-55cb-4431-b07f-49446e79a20e" containerID="d6fc07d15a489b0d472c770c7161ea022cd933ab6f3faefbdb501d234571fdac" exitCode=0
Jun 02 16:58:06.324096 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.323800    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerDied","Data":"d6fc07d15a489b0d472c770c7161ea022cd933ab6f3faefbdb501d234571fdac"}
Jun 02 16:58:06.454514 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.454487    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:58:06.517292 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517188    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert\") pod \"7366d182-55cb-4431-b07f-49446e79a20e\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") "
Jun 02 16:58:06.517292 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517248    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"7366d182-55cb-4431-b07f-49446e79a20e\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") "
Jun 02 16:58:06.517563 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517308    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location\") pod \"7366d182-55cb-4431-b07f-49446e79a20e\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") "
Jun 02 16:58:06.517563 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517345    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgrgl\" (UniqueName: \"kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl\") pod \"7366d182-55cb-4431-b07f-49446e79a20e\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") "
Jun 02 16:58:06.517563 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517402    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls\") pod \"7366d182-55cb-4431-b07f-49446e79a20e\" (UID: \"7366d182-55cb-4431-b07f-49446e79a20e\") "
Jun 02 16:58:06.517735 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517642    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7366d182-55cb-4431-b07f-49446e79a20e" (UID: "7366d182-55cb-4431-b07f-49446e79a20e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:58:06.517735 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517701    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7366d182-55cb-4431-b07f-49446e79a20e" (UID: "7366d182-55cb-4431-b07f-49446e79a20e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:58:06.517825 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517728    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7366d182-55cb-4431-b07f-49446e79a20e-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:06.517825 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.517754    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "7366d182-55cb-4431-b07f-49446e79a20e" (UID: "7366d182-55cb-4431-b07f-49446e79a20e"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:58:06.519716 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.519681    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7366d182-55cb-4431-b07f-49446e79a20e" (UID: "7366d182-55cb-4431-b07f-49446e79a20e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:58:06.519850 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.519718    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl" (OuterVolumeSpecName: "kube-api-access-jgrgl") pod "7366d182-55cb-4431-b07f-49446e79a20e" (UID: "7366d182-55cb-4431-b07f-49446e79a20e"). InnerVolumeSpecName "kube-api-access-jgrgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:58:06.618599 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.618564    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-cabundle-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:06.618775 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.618612    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7366d182-55cb-4431-b07f-49446e79a20e-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:06.618775 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.618623    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrgl\" (UniqueName: \"kubernetes.io/projected/7366d182-55cb-4431-b07f-49446e79a20e-kube-api-access-jgrgl\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:06.618775 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:06.618633    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7366d182-55cb-4431-b07f-49446e79a20e-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:07.329269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.329241    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"
Jun 02 16:58:07.329269 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.329248    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d" event={"ID":"7366d182-55cb-4431-b07f-49446e79a20e","Type":"ContainerDied","Data":"2484da3ec6772cc08a3cb035278351a17a2c8a7611a6dcb81a26c5631bac17f1"}
Jun 02 16:58:07.329827 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.329294    2571 scope.go:117] "RemoveContainer" containerID="2daef734dbf6a61f8895f2a029026cd030aac446f8b460794869f01fd546eef7"
Jun 02 16:58:07.330862 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.330840    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/0.log"
Jun 02 16:58:07.330944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.330882    2571 generic.go:358] "Generic (PLEG): container finished" podID="5b935eda-8248-4dc2-88d8-655806c228af" containerID="bd4c6b3bf4f6c3210bda6dacf74f81838db1619463b2bdafcfca3aa13d549d56" exitCode=1
Jun 02 16:58:07.330944 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.330923    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerDied","Data":"bd4c6b3bf4f6c3210bda6dacf74f81838db1619463b2bdafcfca3aa13d549d56"}
Jun 02 16:58:07.338275 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.338257    2571 scope.go:117] "RemoveContainer" containerID="d6fc07d15a489b0d472c770c7161ea022cd933ab6f3faefbdb501d234571fdac"
Jun 02 16:58:07.346336 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.346314    2571 scope.go:117] "RemoveContainer" containerID="130a666da7a577b0501a1ce43ec8661051b3bbb08cba6620d70dc3306007d235"
Jun 02 16:58:07.368453 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.365526    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:58:07.370119 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.370087    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74d9c55579-bqn6d"]
Jun 02 16:58:07.684250 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:07.684166    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7366d182-55cb-4431-b07f-49446e79a20e" path="/var/lib/kubelet/pods/7366d182-55cb-4431-b07f-49446e79a20e/volumes"
Jun 02 16:58:08.338358 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:08.338328    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/0.log"
Jun 02 16:58:08.338833 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:08.338450    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerStarted","Data":"9689153d1a4c1a194f1472f8e8454e84436345cf083f64261152597ab576f47f"}
Jun 02 16:58:13.069075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.069031    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:13.070112 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.069293    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer" containerID="cri-o://9689153d1a4c1a194f1472f8e8454e84436345cf083f64261152597ab576f47f" gracePeriod=30
Jun 02 16:58:13.359558 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.359532    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/1.log"
Jun 02 16:58:13.360002 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.359984    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/0.log"
Jun 02 16:58:13.360112 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.360023    2571 generic.go:358] "Generic (PLEG): container finished" podID="5b935eda-8248-4dc2-88d8-655806c228af" containerID="9689153d1a4c1a194f1472f8e8454e84436345cf083f64261152597ab576f47f" exitCode=1
Jun 02 16:58:13.360193 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.360174    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerDied","Data":"9689153d1a4c1a194f1472f8e8454e84436345cf083f64261152597ab576f47f"}
Jun 02 16:58:13.360252 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.360221    2571 scope.go:117] "RemoveContainer" containerID="bd4c6b3bf4f6c3210bda6dacf74f81838db1619463b2bdafcfca3aa13d549d56"
Jun 02 16:58:13.422316 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.422292    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/1.log"
Jun 02 16:58:13.422465 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.422431    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:13.474510 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474467    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"5b935eda-8248-4dc2-88d8-655806c228af\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") "
Jun 02 16:58:13.474510 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474509    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jxpk\" (UniqueName: \"kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk\") pod \"5b935eda-8248-4dc2-88d8-655806c228af\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") "
Jun 02 16:58:13.474750 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474557    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location\") pod \"5b935eda-8248-4dc2-88d8-655806c228af\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") "
Jun 02 16:58:13.474750 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474608    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") pod \"5b935eda-8248-4dc2-88d8-655806c228af\" (UID: \"5b935eda-8248-4dc2-88d8-655806c228af\") "
Jun 02 16:58:13.474904 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474878    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "5b935eda-8248-4dc2-88d8-655806c228af" (UID: "5b935eda-8248-4dc2-88d8-655806c228af"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:58:13.474988 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.474888    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b935eda-8248-4dc2-88d8-655806c228af" (UID: "5b935eda-8248-4dc2-88d8-655806c228af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:58:13.476704 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.476673    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b935eda-8248-4dc2-88d8-655806c228af" (UID: "5b935eda-8248-4dc2-88d8-655806c228af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:58:13.476820 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.476744    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk" (OuterVolumeSpecName: "kube-api-access-4jxpk") pod "5b935eda-8248-4dc2-88d8-655806c228af" (UID: "5b935eda-8248-4dc2-88d8-655806c228af"). InnerVolumeSpecName "kube-api-access-4jxpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:58:13.575743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.575682    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b935eda-8248-4dc2-88d8-655806c228af-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:13.575743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.575734    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxpk\" (UniqueName: \"kubernetes.io/projected/5b935eda-8248-4dc2-88d8-655806c228af-kube-api-access-4jxpk\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:13.575743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.575749    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b935eda-8248-4dc2-88d8-655806c228af-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:13.575743 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:13.575759    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b935eda-8248-4dc2-88d8-655806c228af-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:58:14.162252 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162216    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:58:14.162644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162609    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container"
Jun 02 16:58:14.162644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162621    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container"
Jun 02 16:58:14.162644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162629    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kube-rbac-proxy"
Jun 02 16:58:14.162644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162635    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kube-rbac-proxy"
Jun 02 16:58:14.162644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162643    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162649    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162664    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162669    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162678    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162683    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162745    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kserve-container"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162752    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162758    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="7366d182-55cb-4431-b07f-49446e79a20e" containerName="kube-rbac-proxy"
Jun 02 16:58:14.162818 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.162764    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b935eda-8248-4dc2-88d8-655806c228af" containerName="storage-initializer"
Jun 02 16:58:14.167331 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.167310    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.169493 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.169465    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jun 02 16:58:14.169605 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.169524    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jun 02 16:58:14.169605 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.169531    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 02 16:58:14.173574 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.173549    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:58:14.282007 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.281964    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.282192 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.282033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvnt\" (UniqueName: \"kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.282192 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.282058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.282192 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.282085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.282192 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.282144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.364873 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.364846    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt_5b935eda-8248-4dc2-88d8-655806c228af/storage-initializer/1.log"
Jun 02 16:58:14.365063 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.364947    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt" event={"ID":"5b935eda-8248-4dc2-88d8-655806c228af","Type":"ContainerDied","Data":"c7ae6ac127044facef668185d87f8013b0a99a102fe65616cd3bf74c72c970c7"}
Jun 02 16:58:14.365063 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.364986    2571 scope.go:117] "RemoveContainer" containerID="9689153d1a4c1a194f1472f8e8454e84436345cf083f64261152597ab576f47f"
Jun 02 16:58:14.365063 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.365002    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"
Jun 02 16:58:14.383520 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383491    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.383657 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383548    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvnt\" (UniqueName: \"kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.383657 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383575    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.383657 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383615    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.383657 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383652    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.384022 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.383998    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.384154 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.384135    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.384313 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.384290    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.386869 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.386844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.391550 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.391528    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvnt\" (UniqueName: \"kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.394926 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.394898    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:14.400071 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.400047    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-66cfbfcc4f-gj5wt"]
Jun 02 16:58:14.478604 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.478567    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:14.609853 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:14.609740    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:58:14.612742 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:58:14.612713    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993822c_7366_46c8_af48_4690f2550896.slice/crio-bb2f82a3a988526082ed63ea8786eb9e3e011330279994cb26083f8b11f9445a WatchSource:0}: Error finding container bb2f82a3a988526082ed63ea8786eb9e3e011330279994cb26083f8b11f9445a: Status 404 returned error can't find the container with id bb2f82a3a988526082ed63ea8786eb9e3e011330279994cb26083f8b11f9445a
Jun 02 16:58:15.370953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:15.370914    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerStarted","Data":"4e95adc23909a1f3960398a8dddddc0cae92414c9233696c54bc013295f38569"}
Jun 02 16:58:15.370953 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:15.370955    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerStarted","Data":"bb2f82a3a988526082ed63ea8786eb9e3e011330279994cb26083f8b11f9445a"}
Jun 02 16:58:15.684662 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:15.684581    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b935eda-8248-4dc2-88d8-655806c228af" path="/var/lib/kubelet/pods/5b935eda-8248-4dc2-88d8-655806c228af/volumes"
Jun 02 16:58:16.375612 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:16.375571    2571 generic.go:358] "Generic (PLEG): container finished" podID="f993822c-7366-46c8-af48-4690f2550896" containerID="4e95adc23909a1f3960398a8dddddc0cae92414c9233696c54bc013295f38569" exitCode=0
Jun 02 16:58:16.376026 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:16.375661    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerDied","Data":"4e95adc23909a1f3960398a8dddddc0cae92414c9233696c54bc013295f38569"}
Jun 02 16:58:17.380631 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:17.380593    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerStarted","Data":"6eda2a815601818f3393f4fde13f2ef9465ad3d7b06fcf13dfb0500ec33a8c50"}
Jun 02 16:58:17.380631 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:17.380628    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerStarted","Data":"eb821a5af4a96185262694b1ff9d9aa163a9604fcc47062c382e77f86a3b948e"}
Jun 02 16:58:17.381163 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:17.380712    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:17.400079 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:17.400036    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podStartSLOduration=3.400021361 podStartE2EDuration="3.400021361s" podCreationTimestamp="2026-06-02 16:58:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:58:17.39741097 +0000 UTC m=+4294.272937128" watchObservedRunningTime="2026-06-02 16:58:17.400021361 +0000 UTC m=+4294.275547522"
Jun 02 16:58:18.383848 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:18.383811    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:18.385072 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:18.385043    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:58:19.392601 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:19.392549    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:58:24.396739 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:24.396706    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:58:24.397417 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:24.397382    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:58:34.397337 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:34.397290    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:58:44.398247 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:44.398153    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:58:54.397635 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:58:54.397591    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:59:04.398170 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:04.398128    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:59:14.397475 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:14.397433    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:59:24.398002 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:24.397969    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:59:34.183505 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.183470    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:59:34.184034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.183783    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" containerID="cri-o://eb821a5af4a96185262694b1ff9d9aa163a9604fcc47062c382e77f86a3b948e" gracePeriod=30
Jun 02 16:59:34.184034 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.183808    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kube-rbac-proxy" containerID="cri-o://6eda2a815601818f3393f4fde13f2ef9465ad3d7b06fcf13dfb0500ec33a8c50" gracePeriod=30
Jun 02 16:59:34.393386 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.393332    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jun 02 16:59:34.397670 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.397647    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jun 02 16:59:34.647228 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.647195    2571 generic.go:358] "Generic (PLEG): container finished" podID="f993822c-7366-46c8-af48-4690f2550896" containerID="6eda2a815601818f3393f4fde13f2ef9465ad3d7b06fcf13dfb0500ec33a8c50" exitCode=2
Jun 02 16:59:34.647423 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:34.647242    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerDied","Data":"6eda2a815601818f3393f4fde13f2ef9465ad3d7b06fcf13dfb0500ec33a8c50"}
Jun 02 16:59:35.256344 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.256304    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:35.259942 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.259925    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.261844 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.261820    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jun 02 16:59:35.262078 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.262060    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jun 02 16:59:35.271018 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.270995    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:35.393832 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.393790    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtjv\" (UniqueName: \"kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.394011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.393877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.394011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.393922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.394011 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.393947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495315 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495273    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495320    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495359    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495542 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495462    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtjv\" (UniqueName: \"kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495542 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:59:35.495480    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jun 02 16:59:35.495729 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:59:35.495556    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls podName:d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8 nodeName:}" failed. No retries permitted until 2026-06-02 16:59:35.995538865 +0000 UTC m=+4372.871065028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" (UID: "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jun 02 16:59:35.495778 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.495993 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.495974    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.503223 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.503199    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtjv\" (UniqueName: \"kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:35.999922 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:35.999881    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:36.002246 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.002224    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:36.171237 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.171201    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:36.301959 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.301927    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:36.305876 ip-10-0-131-30 kubenswrapper[2571]: W0602 16:59:36.305848    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27d5e2b_fbdc_411d_9f2e_0bee4409e5f8.slice/crio-b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd WatchSource:0}: Error finding container b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd: Status 404 returned error can't find the container with id b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd
Jun 02 16:59:36.307959 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.307944    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:59:36.656048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.655953    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerStarted","Data":"fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a"}
Jun 02 16:59:36.656048 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:36.655994    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerStarted","Data":"b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd"}
Jun 02 16:59:38.664189 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.664157    2571 generic.go:358] "Generic (PLEG): container finished" podID="f993822c-7366-46c8-af48-4690f2550896" containerID="eb821a5af4a96185262694b1ff9d9aa163a9604fcc47062c382e77f86a3b948e" exitCode=0
Jun 02 16:59:38.664564 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.664227    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerDied","Data":"eb821a5af4a96185262694b1ff9d9aa163a9604fcc47062c382e77f86a3b948e"}
Jun 02 16:59:38.735578 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.735553    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:59:38.929500 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929407    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqvnt\" (UniqueName: \"kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt\") pod \"f993822c-7366-46c8-af48-4690f2550896\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") "
Jun 02 16:59:38.929658 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929503    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"f993822c-7366-46c8-af48-4690f2550896\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") "
Jun 02 16:59:38.929658 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929536    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls\") pod \"f993822c-7366-46c8-af48-4690f2550896\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") "
Jun 02 16:59:38.929658 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929579    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert\") pod \"f993822c-7366-46c8-af48-4690f2550896\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") "
Jun 02 16:59:38.929658 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929633    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location\") pod \"f993822c-7366-46c8-af48-4690f2550896\" (UID: \"f993822c-7366-46c8-af48-4690f2550896\") "
Jun 02 16:59:38.930008 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929977    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "f993822c-7366-46c8-af48-4690f2550896" (UID: "f993822c-7366-46c8-af48-4690f2550896"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:59:38.930008 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.929986    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f993822c-7366-46c8-af48-4690f2550896" (UID: "f993822c-7366-46c8-af48-4690f2550896"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:59:38.930194 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.930034    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f993822c-7366-46c8-af48-4690f2550896" (UID: "f993822c-7366-46c8-af48-4690f2550896"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:59:38.931718 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.931698    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt" (OuterVolumeSpecName: "kube-api-access-pqvnt") pod "f993822c-7366-46c8-af48-4690f2550896" (UID: "f993822c-7366-46c8-af48-4690f2550896"). InnerVolumeSpecName "kube-api-access-pqvnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:59:38.931792 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:38.931697    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f993822c-7366-46c8-af48-4690f2550896" (UID: "f993822c-7366-46c8-af48-4690f2550896"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:59:39.030665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.030628    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:39.030665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.030659    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f993822c-7366-46c8-af48-4690f2550896-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:39.030665 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.030672    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f993822c-7366-46c8-af48-4690f2550896-cabundle-cert\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:39.030910 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.030682    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f993822c-7366-46c8-af48-4690f2550896-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:39.030910 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.030696    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvnt\" (UniqueName: \"kubernetes.io/projected/f993822c-7366-46c8-af48-4690f2550896-kube-api-access-pqvnt\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:39.669071 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.669031    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx" event={"ID":"f993822c-7366-46c8-af48-4690f2550896","Type":"ContainerDied","Data":"bb2f82a3a988526082ed63ea8786eb9e3e011330279994cb26083f8b11f9445a"}
Jun 02 16:59:39.669071 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.669076    2571 scope.go:117] "RemoveContainer" containerID="6eda2a815601818f3393f4fde13f2ef9465ad3d7b06fcf13dfb0500ec33a8c50"
Jun 02 16:59:39.669658 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.669074    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"
Jun 02 16:59:39.677220 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.677198    2571 scope.go:117] "RemoveContainer" containerID="eb821a5af4a96185262694b1ff9d9aa163a9604fcc47062c382e77f86a3b948e"
Jun 02 16:59:39.685407 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.685383    2571 scope.go:117] "RemoveContainer" containerID="4e95adc23909a1f3960398a8dddddc0cae92414c9233696c54bc013295f38569"
Jun 02 16:59:39.690043 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.690020    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:59:39.693075 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:39.693052    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-668dbb4b58-mzcmx"]
Jun 02 16:59:40.673593 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:40.673572    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/0.log"
Jun 02 16:59:40.673972 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:40.673606    2571 generic.go:358] "Generic (PLEG): container finished" podID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" containerID="fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a" exitCode=1
Jun 02 16:59:40.673972 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:40.673672    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerDied","Data":"fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a"}
Jun 02 16:59:41.680084 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:41.680053    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/0.log"
Jun 02 16:59:41.683734 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:41.683709    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f993822c-7366-46c8-af48-4690f2550896" path="/var/lib/kubelet/pods/f993822c-7366-46c8-af48-4690f2550896/volumes"
Jun 02 16:59:41.684056 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:41.684039    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerStarted","Data":"10e2c674ae62033eb81af3958e7308feef0ee6c54305bbf53ae7a3472a58fcfa"}
Jun 02 16:59:44.690111 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690078    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/1.log"
Jun 02 16:59:44.690576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690428    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/0.log"
Jun 02 16:59:44.690576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690464    2571 generic.go:358] "Generic (PLEG): container finished" podID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" containerID="10e2c674ae62033eb81af3958e7308feef0ee6c54305bbf53ae7a3472a58fcfa" exitCode=1
Jun 02 16:59:44.690576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690518    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerDied","Data":"10e2c674ae62033eb81af3958e7308feef0ee6c54305bbf53ae7a3472a58fcfa"}
Jun 02 16:59:44.690576 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690554    2571 scope.go:117] "RemoveContainer" containerID="fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a"
Jun 02 16:59:44.690970 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:44.690955    2571 scope.go:117] "RemoveContainer" containerID="fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a"
Jun 02 16:59:44.701074 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:59:44.701044    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_kserve-ci-e2e-test_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8_0 in pod sandbox b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd from index: no such id: 'fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a'" containerID="fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a"
Jun 02 16:59:44.701164 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:59:44.701113    2571 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_kserve-ci-e2e-test_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8_0 in pod sandbox b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd from index: no such id: 'fd47339af01f5301c98000407bf7bc1dd1c462e04bc2aa71683679749852d31a'; Skipping pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_kserve-ci-e2e-test(d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8)\"" logger="UnhandledError"
Jun 02 16:59:44.702485 ip-10-0-131-30 kubenswrapper[2571]: E0602 16:59:44.702462    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_kserve-ci-e2e-test(d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" podUID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8"
Jun 02 16:59:45.234385 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.234327    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:45.695266 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.695183    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/1.log"
Jun 02 16:59:45.829481 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.829460    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/1.log"
Jun 02 16:59:45.829640 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.829536    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:45.882324 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882293    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location\") pod \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") "
Jun 02 16:59:45.882518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882344    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") "
Jun 02 16:59:45.882518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882412    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhtjv\" (UniqueName: \"kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv\") pod \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") "
Jun 02 16:59:45.882518 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882440    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") pod \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\" (UID: \"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8\") "
Jun 02 16:59:45.882644 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882590    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" (UID: "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:59:45.882709 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882683    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kserve-provision-location\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:45.882812 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.882784    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" (UID: "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:59:45.884555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.884529    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" (UID: "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:59:45.884555 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.884550    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv" (OuterVolumeSpecName: "kube-api-access-xhtjv") pod "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" (UID: "d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8"). InnerVolumeSpecName "kube-api-access-xhtjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:59:45.983806 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.983768    2571 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:45.983806 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.983804    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtjv\" (UniqueName: \"kubernetes.io/projected/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-kube-api-access-xhtjv\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:45.984016 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:45.983815    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8-proxy-tls\") on node \"ip-10-0-131-30.ec2.internal\" DevicePath \"\""
Jun 02 16:59:46.699929 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.699901    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg_d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/storage-initializer/1.log"
Jun 02 16:59:46.700354 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.700035    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg" event={"ID":"d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8","Type":"ContainerDied","Data":"b0190600cd517a3a7df8b69a39619f9c5a034c9a1cfdc5a5853de3b11f07a1fd"}
Jun 02 16:59:46.700354 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.700064    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"
Jun 02 16:59:46.700354 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.700083    2571 scope.go:117] "RemoveContainer" containerID="10e2c674ae62033eb81af3958e7308feef0ee6c54305bbf53ae7a3472a58fcfa"
Jun 02 16:59:46.734032 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.734005    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:46.736989 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:46.736965    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5cfdc5b88d-c6gmg"]
Jun 02 16:59:47.683890 ip-10-0-131-30 kubenswrapper[2571]: I0602 16:59:47.683856    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" path="/var/lib/kubelet/pods/d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8/volumes"
Jun 02 17:00:21.628944 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:21.628900    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5p997_3bcb6f1f-892d-413e-b840-ea5186b36cf6/global-pull-secret-syncer/0.log"
Jun 02 17:00:21.787863 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:21.787829    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kj5mf_d9111172-9d4c-4285-9e39-ba0c9a637533/konnectivity-agent/0.log"
Jun 02 17:00:21.919935 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:21.919844    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-30.ec2.internal_3240da12bbab1be9fe19581d21610356/haproxy/0.log"
Jun 02 17:00:25.777587 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:25.777537    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-69b6cbdb88-qncnd_9601e117-4d6e-4a2b-ba90-2b0fc86e4a6a/cluster-monitoring-operator/0.log"
Jun 02 17:00:25.800496 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:25.800466    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-state-metrics/0.log"
Jun 02 17:00:25.817048 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:25.817011    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-rbac-proxy-main/0.log"
Jun 02 17:00:25.834829 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:25.834798    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-4lqzx_5a08ddb5-3017-476a-9257-e4acc59fa0c3/kube-rbac-proxy-self/0.log"
Jun 02 17:00:26.039464 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.039381    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/node-exporter/0.log"
Jun 02 17:00:26.059195 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.059161    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/kube-rbac-proxy/0.log"
Jun 02 17:00:26.077194 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.077169    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzv6p_e317b143-129b-4b87-b378-459f58efb816/init-textfile/0.log"
Jun 02 17:00:26.100333 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.100284    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/kube-rbac-proxy-main/0.log"
Jun 02 17:00:26.122172 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.122137    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/kube-rbac-proxy-self/0.log"
Jun 02 17:00:26.147890 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.147848    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-wqgmk_cdf11c0b-146e-4805-8b4e-a231d2033047/openshift-state-metrics/0.log"
Jun 02 17:00:26.524302 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.524271    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/thanos-query/0.log"
Jun 02 17:00:26.546379 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.546331    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/kube-rbac-proxy-web/0.log"
Jun 02 17:00:26.568834 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.568797    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/kube-rbac-proxy/0.log"
Jun 02 17:00:26.589129 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.589095    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/prom-label-proxy/0.log"
Jun 02 17:00:26.611443 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.611406    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/kube-rbac-proxy-rules/0.log"
Jun 02 17:00:26.631499 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:26.631474    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75d476cbb8-smv6x_b5546459-be11-4809-af8a-72d74619c686/kube-rbac-proxy-metrics/0.log"
Jun 02 17:00:28.334661 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.334628    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76fc654ccf-xnzsv_75e12a91-4ab5-4212-8307-69a891bc4b34/console/0.log"
Jun 02 17:00:28.747655 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.747618    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"]
Jun 02 17:00:28.747971 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.747959    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kube-rbac-proxy"
Jun 02 17:00:28.748016 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.747972    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kube-rbac-proxy"
Jun 02 17:00:28.748016 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.747988    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" containerName="storage-initializer"
Jun 02 17:00:28.748016 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.747994    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" containerName="storage-initializer"
Jun 02 17:00:28.748016 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748001    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="storage-initializer"
Jun 02 17:00:28.748016 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748007    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="storage-initializer"
Jun 02 17:00:28.748176 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748022    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container"
Jun 02 17:00:28.748176 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748027    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container"
Jun 02 17:00:28.748176 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748081    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kserve-container"
Jun 02 17:00:28.748176 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748091    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="d27d5e2b-fbdc-411d-9f2e-0bee4409e5f8" containerName="storage-initializer"
Jun 02 17:00:28.748176 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.748098    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="f993822c-7366-46c8-af48-4690f2550896" containerName="kube-rbac-proxy"
Jun 02 17:00:28.751209 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.751192    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.753549 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.753528    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpxbm\"/\"kube-root-ca.crt\""
Jun 02 17:00:28.754271 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.754247    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpxbm\"/\"default-dockercfg-kvlpp\""
Jun 02 17:00:28.754393 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.754275    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpxbm\"/\"openshift-service-ca.crt\""
Jun 02 17:00:28.759326 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.759306    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"]
Jun 02 17:00:28.768705 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.768665    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7598fd5bd-mxr8d_299c6efd-0128-40ba-a067-fc0a3f51e7a6/volume-data-source-validator/0.log"
Jun 02 17:00:28.864057 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.864019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-podres\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.864260 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.864097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-lib-modules\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.864260 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.864147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-proc\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.864260 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.864198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-sys\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.864260 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.864249    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz49p\" (UniqueName: \"kubernetes.io/projected/324829c0-9699-47eb-8688-caf61f9c5f11-kube-api-access-zz49p\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.964900 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.964862    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-proc\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.964900 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.964901    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-sys\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.964940    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zz49p\" (UniqueName: \"kubernetes.io/projected/324829c0-9699-47eb-8688-caf61f9c5f11-kube-api-access-zz49p\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.964992    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-podres\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.965007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-sys\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.965011    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-proc\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.965055    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-lib-modules\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965160 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.965129    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-podres\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.965424 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.965205    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/324829c0-9699-47eb-8688-caf61f9c5f11-lib-modules\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:28.972734 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:28.972709    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz49p\" (UniqueName: \"kubernetes.io/projected/324829c0-9699-47eb-8688-caf61f9c5f11-kube-api-access-zz49p\") pod \"perf-node-gather-daemonset-grp72\" (UID: \"324829c0-9699-47eb-8688-caf61f9c5f11\") " pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:29.062353 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.062256    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:29.188793 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.188758    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"]
Jun 02 17:00:29.192240 ip-10-0-131-30 kubenswrapper[2571]: W0602 17:00:29.192203    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod324829c0_9699_47eb_8688_caf61f9c5f11.slice/crio-7e7fcf7faaec5611f8dc5e5a57499ddd0542a217895f9398d30b5543b89483ee WatchSource:0}: Error finding container 7e7fcf7faaec5611f8dc5e5a57499ddd0542a217895f9398d30b5543b89483ee: Status 404 returned error can't find the container with id 7e7fcf7faaec5611f8dc5e5a57499ddd0542a217895f9398d30b5543b89483ee
Jun 02 17:00:29.440855 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.440814    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v22gh_b0d74645-7b90-430f-af77-f554a9418180/dns/0.log"
Jun 02 17:00:29.459869 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.459843    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v22gh_b0d74645-7b90-430f-af77-f554a9418180/kube-rbac-proxy/0.log"
Jun 02 17:00:29.543792 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.543757    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r4wc4_ea971506-fe66-418b-a884-855d031a66aa/dns-node-resolver/0.log"
Jun 02 17:00:29.849594 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.849560    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72" event={"ID":"324829c0-9699-47eb-8688-caf61f9c5f11","Type":"ContainerStarted","Data":"4f657354443a8d546eaefdc159a98798623f44d15df4fc5664508937db4e5865"}
Jun 02 17:00:29.849594 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.849599    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72" event={"ID":"324829c0-9699-47eb-8688-caf61f9c5f11","Type":"ContainerStarted","Data":"7e7fcf7faaec5611f8dc5e5a57499ddd0542a217895f9398d30b5543b89483ee"}
Jun 02 17:00:29.849819 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.849646    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:29.865784 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:29.865734    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72" podStartSLOduration=1.865719637 podStartE2EDuration="1.865719637s" podCreationTimestamp="2026-06-02 17:00:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:00:29.863776111 +0000 UTC m=+4426.739302272" watchObservedRunningTime="2026-06-02 17:00:29.865719637 +0000 UTC m=+4426.741245797"
Jun 02 17:00:30.020949 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:30.020920    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjg4n_6e1a921c-9997-4cd6-937f-a1a1a2967260/node-ca/0.log"
Jun 02 17:00:30.666284 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:30.666253    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bc549f7b9-ncs76_34318a4f-dd28-449c-ae48-84eec2db293b/router/0.log"
Jun 02 17:00:30.974752 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:30.974724    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5wfsn_e3b6c156-e1a0-4290-9a47-55dd7016276e/serve-healthcheck-canary/0.log"
Jun 02 17:00:31.366342 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:31.366246    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-788bf58df5-c2bl2_2db81b66-10f3-4fc9-a130-484b26440831/insights-operator/0.log"
Jun 02 17:00:31.369258 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:31.369229    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-788bf58df5-c2bl2_2db81b66-10f3-4fc9-a130-484b26440831/insights-operator/1.log"
Jun 02 17:00:31.498804 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:31.498775    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdd6d_3ae18564-cb3d-4791-95d7-5e0f3b1e3822/kube-rbac-proxy/0.log"
Jun 02 17:00:31.516235 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:31.516208    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdd6d_3ae18564-cb3d-4791-95d7-5e0f3b1e3822/exporter/0.log"
Jun 02 17:00:31.533717 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:31.533689    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdd6d_3ae18564-cb3d-4791-95d7-5e0f3b1e3822/extractor/0.log"
Jun 02 17:00:33.508921 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:33.508888    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7b649f477f-8jx6t_f6fde234-c306-4280-95a5-f41746e4665a/manager/0.log"
Jun 02 17:00:33.527196 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:33.527168    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-9fj7n_140e5bfd-cad1-4b86-935f-0baa9fd73d4f/manager/0.log"
Jun 02 17:00:33.545286 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:33.545263    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-ggjmn_41d45046-1219-4117-a3b5-19b3c7d8f86d/server/0.log"
Jun 02 17:00:33.889702 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:33.889614    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-5c88b85bb7-lrj4v_c3f163e2-c7b9-4a83-a06c-fd581f9b0733/seaweedfs-tls-custom/0.log"
Jun 02 17:00:33.910739 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:33.910710    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-4l4xg_345ef462-6ddc-498f-a6ff-0c33412fd7a3/seaweedfs-tls-serving/0.log"
Jun 02 17:00:35.862581 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:35.862556    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpxbm/perf-node-gather-daemonset-grp72"
Jun 02 17:00:37.462959 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:37.462921    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-96c6f749-ktn6r_55339f68-4ed2-4754-a3c5-a78690703a70/migrator/0.log"
Jun 02 17:00:37.480644 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:37.480616    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-96c6f749-ktn6r_55339f68-4ed2-4754-a3c5-a78690703a70/graceful-termination/0.log"
Jun 02 17:00:37.867231 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:37.867152    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-96ddc787c-pn2bq_8f118d48-4f19-4dc1-a22a-b456f271c48c/kube-storage-version-migrator-operator/1.log"
Jun 02 17:00:37.869443 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:37.869399    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-96ddc787c-pn2bq_8f118d48-4f19-4dc1-a22a-b456f271c48c/kube-storage-version-migrator-operator/0.log"
Jun 02 17:00:39.003815 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.003783    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/kube-multus-additional-cni-plugins/0.log"
Jun 02 17:00:39.023874 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.023847    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/egress-router-binary-copy/0.log"
Jun 02 17:00:39.041514 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.041489    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/cni-plugins/0.log"
Jun 02 17:00:39.059283 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.059249    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/bond-cni-plugin/0.log"
Jun 02 17:00:39.076694 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.076616    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/routeoverride-cni/0.log"
Jun 02 17:00:39.093599 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.093571    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/whereabouts-cni-bincopy/0.log"
Jun 02 17:00:39.110257 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.110228    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xzf5d_a71f5380-9081-4609-8da0-42da280a618d/whereabouts-cni/0.log"
Jun 02 17:00:39.281968 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.281933    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gbcbz_51da022a-4320-403f-bee1-05113613afb4/kube-multus/0.log"
Jun 02 17:00:39.424875 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.424774    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qqr9_e4ecbb10-dcd6-4827-93fc-7ffc14508a87/network-metrics-daemon/0.log"
Jun 02 17:00:39.438598 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:39.438573    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qqr9_e4ecbb10-dcd6-4827-93fc-7ffc14508a87/kube-rbac-proxy/0.log"
Jun 02 17:00:40.199687 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.199655    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-controller/0.log"
Jun 02 17:00:40.213602 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.213569    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/0.log"
Jun 02 17:00:40.253266 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.253239    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovn-acl-logging/1.log"
Jun 02 17:00:40.274043 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.273964    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/kube-rbac-proxy-node/0.log"
Jun 02 17:00:40.293413 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.293384    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 17:00:40.307909 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.307878    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/northd/0.log"
Jun 02 17:00:40.326165 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.326140    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/nbdb/0.log"
Jun 02 17:00:40.344501 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.344477    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/sbdb/0.log"
Jun 02 17:00:40.529624 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:40.529524    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9st6m_631ad240-3383-480d-ab82-bde14902804e/ovnkube-controller/0.log"
Jun 02 17:00:42.002330 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:42.002299    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wmkbn_68584f01-5fd0-42c0-a157-392408a5f2cb/network-check-target-container/0.log"
Jun 02 17:00:42.883642 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:42.883611    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b5lv9_3b4a915b-6de0-4eac-8cf7-a4d48060b2e4/iptables-alerter/0.log"
Jun 02 17:00:43.550862 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:43.550832    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7wbzp_3e5af621-7c9a-437e-9195-e97b1fa0d25e/tuned/0.log"
Jun 02 17:00:45.182665 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:45.182633    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-755bbf9bfb-wzg84_6fa6ba27-492c-4831-be65-53e14d4d825d/cluster-samples-operator/0.log"
Jun 02 17:00:45.196965 ip-10-0-131-30 kubenswrapper[2571]: I0602 17:00:45.196928    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-755bbf9bfb-wzg84_6fa6ba27-492c-4831-be65-53e14d4d825d/cluster-samples-operator-watch/0.log"