Apr 17 16:28:43.579890 ip-10-0-141-140 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Apr 17 16:28:43.579909 ip-10-0-141-140 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Apr 17 16:28:43.579920 ip-10-0-141-140 systemd[1]: kubelet.service: Failed with result 'resources'.
Apr 17 16:28:43.580322 ip-10-0-141-140 systemd[1]: Failed to start Kubernetes Kubelet.
Apr 17 16:28:53.702753 ip-10-0-141-140 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Apr 17 16:28:53.702771 ip-10-0-141-140 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0b790b4bc14a44d68f3af592cbb3ef41 --
Apr 17 16:31:12.622046 ip-10-0-141-140 systemd[1]: Starting Kubernetes Kubelet...
Apr 17 16:31:13.140225 ip-10-0-141-140 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 17 16:31:13.140225 ip-10-0-141-140 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 17 16:31:13.140225 ip-10-0-141-140 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 17 16:31:13.140225 ip-10-0-141-140 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 17 16:31:13.140225 ip-10-0-141-140 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 17 16:31:13.142105 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.142013    2579 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 17 16:31:13.145207 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145191    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 17 16:31:13.145207 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145207    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145212    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145217    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145222    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145227    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145230    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145233    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145236    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145239    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145242    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145245    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145248    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145251    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145254    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145257    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145260    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145263    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145266    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145268    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145271    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 17 16:31:13.145272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145274    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145277    2579 feature_gate.go:328] unrecognized feature gate: Example
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145280    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145283    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145286    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145288    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145291    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145293    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145296    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145298    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145301    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145303    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145306    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145309    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145311    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145314    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145316    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145320    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145322    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145325    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 17 16:31:13.145921 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145328    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145330    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145333    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145335    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145338    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145340    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145342    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145345    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145349    2579 feature_gate.go:328] unrecognized feature gate: Example2
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145351    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145354    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145356    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145359    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145361    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145365    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145368    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145371    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145374    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145376    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145379    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 17 16:31:13.146590 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145381    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145384    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145387    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145391    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145394    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145396    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145399    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145402    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145405    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145408    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145411    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145414    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145416    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145418    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145422    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145424    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145427    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145429    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145432    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 17 16:31:13.147104 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145435    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 17 16:31:13.147574 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145437    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 17 16:31:13.147574 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145439    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 17 16:31:13.147574 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145442    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 17 16:31:13.147574 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145445    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 17 16:31:13.147574 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.145447    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 17 16:31:13.149449 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149434    2579 feature_gate.go:328] unrecognized feature gate: Example2
Apr 17 16:31:13.149449 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149448    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149452    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149455    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149458    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149464    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149467    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149470    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149473    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149476    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149480    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149484    2579 feature_gate.go:328] unrecognized feature gate: Example
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149487    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149490    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149493    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149496    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149499    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149503    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149508    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149511    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 17 16:31:13.149509 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149514    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149517    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149521    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149523    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149526    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149529    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149531    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149534    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149538    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149543    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149545    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149548    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149550    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149553    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149556    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149560    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149564    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149566    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149569    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 17 16:31:13.149989 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149572    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149574    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149577    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149582    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149585    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149587    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149590    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149593    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149595    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149598    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149602    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149606    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149608    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149611    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149613    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149617    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149623    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149625    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149628    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149631    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 17 16:31:13.150501 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149634    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149637    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149639    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149642    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149645    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149647    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149650    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149652    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149655    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149660    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149663    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149665    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149668    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149670    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149673    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149676    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149693    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149697    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149700    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 17 16:31:13.151005 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149703    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149710    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149714    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149717    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149720    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149724    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149727    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.149730    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 17 16:31:13.151466 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.149954    2579 flags.go:64] FLAG: --address="0.0.0.0"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.149972    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151493    2579 flags.go:64] FLAG: --anonymous-auth="true"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151509    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151514    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151517    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151522    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151527    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151530    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151533    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151536    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151540    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151543    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151546    2579 flags.go:64] FLAG: --cgroup-root=""
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151549    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151552    2579 flags.go:64] FLAG: --client-ca-file=""
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151555    2579 flags.go:64] FLAG: --cloud-config=""
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151558    2579 flags.go:64] FLAG: --cloud-provider="external"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151561    2579 flags.go:64] FLAG: --cluster-dns="[]"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151566    2579 flags.go:64] FLAG: --cluster-domain=""
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151569    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151572    2579 flags.go:64] FLAG: --config-dir=""
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151575    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151579    2579 flags.go:64] FLAG: --container-log-max-files="5"
Apr 17 16:31:13.151670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151583    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151586    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151589    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151592    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151595    2579 flags.go:64] FLAG: --contention-profiling="false"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151598    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151601    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151604    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151608    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151613    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151617    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151620    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151623    2579 flags.go:64] FLAG: --enable-load-reader="false"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151626    2579 flags.go:64] FLAG: --enable-server="true"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151629    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151634    2579 flags.go:64] FLAG: --event-burst="100"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151638    2579 flags.go:64] FLAG: --event-qps="50"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151641    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151644    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151647    2579 flags.go:64] FLAG: --eviction-hard=""
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151651    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151654    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151657    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151660    2579 flags.go:64] FLAG: --eviction-soft=""
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151663    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 17 16:31:13.152268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151666    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151669    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151672    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151675    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151678    2579 flags.go:64] FLAG: --fail-swap-on="true"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151681    2579 flags.go:64] FLAG: --feature-gates=""
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151685    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151688    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151691    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151695    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151698    2579 flags.go:64] FLAG: --healthz-port="10248"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151701    2579 flags.go:64] FLAG: --help="false"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151704    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151708    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151711    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151714    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151717    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151722    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151725    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151728    2579 flags.go:64] FLAG: --image-service-endpoint=""
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151731    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151734    2579 flags.go:64] FLAG: --kube-api-burst="100"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151737    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151740    2579 flags.go:64] FLAG: --kube-api-qps="50"
Apr 17 16:31:13.152869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151743    2579 flags.go:64] FLAG: --kube-reserved=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151746    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151749    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151752    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151755    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151771    2579 flags.go:64] FLAG: --lock-file=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151775    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151778    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151781    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151787    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151790    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151793    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151796    2579 flags.go:64] FLAG: --logging-format="text"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151800    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151803    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151806    2579 flags.go:64] FLAG: --manifest-url=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151809    2579 flags.go:64] FLAG: --manifest-url-header=""
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151814    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151817    2579 flags.go:64] FLAG: --max-open-files="1000000"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151821    2579 flags.go:64] FLAG: --max-pods="110"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151824    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151827    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151830    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151834    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151836    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 17 16:31:13.153436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151840    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151843    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151852    2579 flags.go:64] FLAG: --node-status-max-images="50"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151855    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151858    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151861    2579 flags.go:64] FLAG: --pod-cidr=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151864    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151868    2579 flags.go:64] FLAG: --pod-manifest-path=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151871    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151874    2579 flags.go:64] FLAG: --pods-per-core="0"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151878    2579 flags.go:64] FLAG: --port="10250"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151881    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151884    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e114ee25936f342e"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151887    2579 flags.go:64] FLAG: --qos-reserved=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151890    2579 flags.go:64] FLAG: --read-only-port="10255"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151893    2579 flags.go:64] FLAG: --register-node="true"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151896    2579 flags.go:64] FLAG: --register-schedulable="true"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151899    2579 flags.go:64] FLAG: --register-with-taints=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151903    2579 flags.go:64] FLAG: --registry-burst="10"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151906    2579 flags.go:64] FLAG: --registry-qps="5"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151909    2579 flags.go:64] FLAG: --reserved-cpus=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151912    2579 flags.go:64] FLAG: --reserved-memory=""
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151916    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151919    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151922    2579 flags.go:64] FLAG: --rotate-certificates="false"
Apr 17 16:31:13.154066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151925    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151928    2579 flags.go:64] FLAG: --runonce="false"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151932    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151935    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151938    2579 flags.go:64] FLAG: --seccomp-default="false"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151941    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151943    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151947    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151950    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151954    2579 flags.go:64] FLAG: --storage-driver-password="root"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151957    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151960    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151963    2579 flags.go:64] FLAG: --storage-driver-user="root"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151966    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151969    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151972    2579 flags.go:64] FLAG: --system-cgroups=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151974    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151981    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151984    2579 flags.go:64] FLAG: --tls-cert-file=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151987    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151992    2579 flags.go:64] FLAG: --tls-min-version=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151995    2579 flags.go:64] FLAG: --tls-private-key-file=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.151998    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152001    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152005    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 17 16:31:13.154656 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152008    2579 flags.go:64] FLAG: --v="2"
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152012    2579 flags.go:64] FLAG: --version="false"
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152016    2579 flags.go:64] FLAG: --vmodule=""
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152020    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.152024    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152121    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152126    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152129    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152132    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152136    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152138    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152141    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152144    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152148    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152154    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152158    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152161    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152164    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152167    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152170    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 17 16:31:13.155278 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152173    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152176    2579 feature_gate.go:328] unrecognized feature gate: Example
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152179    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152181    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152184    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152187    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152189    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152192    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152195    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152197    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152200    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152203    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152205    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152208    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152210    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152213    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152215    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152218    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152221    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152223    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 17 16:31:13.155856 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152226    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152229    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152232    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152234    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152237    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152240    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152243    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152245    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152249    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152251    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152254    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152256    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152259    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152262    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152264    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152267    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152270    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152272    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152275    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152278    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 17 16:31:13.156369 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152281    2579 feature_gate.go:328] unrecognized feature gate: Example2
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152283    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152288    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152291    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152294    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152297    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152299    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152302    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152304    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152307    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152310    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152312    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152315    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152317    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152320    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152323    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152326    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152328    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152331    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 17 16:31:13.156869 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152334    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152337    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152339    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152342    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152345    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152347    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152350    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152352    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152355    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152357    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152360    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.152362    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 17 16:31:13.157321 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.153175    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 17 16:31:13.160907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.160886    2579 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 17 16:31:13.160948 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.160908    2579 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160955    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160960    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160963    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160966    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160969    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160972    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160975    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160978    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160980    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 17 16:31:13.160980 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160983    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160988    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160992    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160995    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.160998    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161001    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161004    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161006    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161009    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161012    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161014    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161017    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161020    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161023    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161026    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161028    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161031    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161033    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161036    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 17 16:31:13.161218 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161038    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161041    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161044    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161047    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161050    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161052    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161055    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161058    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161062    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161065    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161068    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161070    2579 feature_gate.go:328] unrecognized feature gate: Example2
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161073    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161075    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161078    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161081    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161084    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161086    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161089    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161092    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 17 16:31:13.161683 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161095    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161097    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161100    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161103    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161105    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161108    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161110    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161113    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161116    2579 feature_gate.go:328] unrecognized feature gate: Example
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161118    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161121    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161124    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161126    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161129    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161131    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161134    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161137    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161139    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161142    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161145    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 17 16:31:13.162245 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161147    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161150    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161153    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161155    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161158    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161160    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161163    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161166    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161169    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161172    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161174    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161177    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161179    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161182    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161184    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161187    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161189    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 17 16:31:13.162784 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161192    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.161197    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161300    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161304    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161307    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161310    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161313    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161316    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161320    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161324    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161327    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161329    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161333    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161335    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161338    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 17 16:31:13.163198 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161341    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161344    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161347    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161350    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161353    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161356    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161358    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161362    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161365    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161368    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161371    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161374    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161377    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161379    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161382    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161384    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161387    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161389    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161392    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161395    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 17 16:31:13.163583 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161397    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161400    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161402    2579 feature_gate.go:328] unrecognized feature gate: Example2
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161405    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161409    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161412    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161414    2579 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161417    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161419    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161422    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161425    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161428    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161431    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161433    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161436    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161438    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161441    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161443    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161446    2579 feature_gate.go:328] unrecognized feature gate: Example
Apr 17 16:31:13.164090 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161449    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161452    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161455    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161457    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161460    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161463    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161465    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161468    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161470    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161473    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161475    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161478    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161480    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161483    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161485    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161488    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161490    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161493    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161495    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161498    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 17 16:31:13.164553 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161500    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161503    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161505    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161508    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161511    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161513    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161516    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161518    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161521    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161523    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161526    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161528    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161531    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:13.161533    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.161538    2579 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 17 16:31:13.165049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.162371    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 17 16:31:13.165418 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.164674    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 17 16:31:13.165751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.165738    2579 server.go:1019] "Starting client certificate rotation"
Apr 17 16:31:13.165813 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.165788    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 17 16:31:13.165905 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.165834    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 17 16:31:13.195725 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.195699    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 17 16:31:13.198390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.198373    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 17 16:31:13.209123 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.209100    2579 log.go:25] "Validated CRI v1 runtime API"
Apr 17 16:31:13.215667 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.215651    2579 log.go:25] "Validated CRI v1 image API"
Apr 17 16:31:13.216998 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.216981    2579 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 17 16:31:13.221613 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.221592    2579 fs.go:135] Filesystem UUIDs: map[72138675-ed3c-485a-b69a-430549bba05b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ded4822a-4fb0-4447-90f0-01336288deeb:/dev/nvme0n1p3]
Apr 17 16:31:13.221672 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.221612    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 17 16:31:13.222981 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.222965    2579 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 17 16:31:13.227686 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.227580    2579 manager.go:217] Machine: {Timestamp:2026-04-17 16:31:13.225208486 +0000 UTC m=+0.465866914 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098634 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec290a698faa9242ff34ced2292a613c SystemUUID:ec290a69-8faa-9242-ff34-ced2292a613c BootID:0b790b4b-c14a-44d6-8f3a-f592cbb3ef41 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:77:9b:1a:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:77:9b:1a:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:92:11:22:14:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 17 16:31:13.227686 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.227681    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 17 16:31:13.227847 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.227756    2579 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 17 16:31:13.229708 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.229684    2579 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 17 16:31:13.229881 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.229710    2579 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 17 16:31:13.229933 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.229891    2579 topology_manager.go:138] "Creating topology manager with none policy"
Apr 17 16:31:13.229933 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.229899    2579 container_manager_linux.go:306] "Creating device plugin manager"
Apr 17 16:31:13.229933 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.229912    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 17 16:31:13.231040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.231030    2579 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 17 16:31:13.232544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.232534    2579 state_mem.go:36] "Initialized new in-memory state store"
Apr 17 16:31:13.232641 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.232631    2579 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 17 16:31:13.235397 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.235388    2579 kubelet.go:491] "Attempting to sync node with API server"
Apr 17 16:31:13.235439 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.235407    2579 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 17 16:31:13.235439 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.235420    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 17 16:31:13.235439 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.235429    2579 kubelet.go:397] "Adding apiserver pod source"
Apr 17 16:31:13.235516 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.235441    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 17 16:31:13.236549 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.236538    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 17 16:31:13.236596 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.236556    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 17 16:31:13.240603 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.240587    2579 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 17 16:31:13.240695 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.240623    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6tkwh"
Apr 17 16:31:13.241999 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.241984    2579 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 17 16:31:13.244525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244512    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244532    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244541    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244549    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244558    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244567    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244575    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244583    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244592    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 17 16:31:13.244608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244601    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 17 16:31:13.244907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244621    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 17 16:31:13.244907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.244634    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 17 16:31:13.245590 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.245579    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 17 16:31:13.245641 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.245594    2579 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 17 16:31:13.247909 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.247876    2579 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 17 16:31:13.248811 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.248778    2579 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 17 16:31:13.248964 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.248944    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6tkwh"
Apr 17 16:31:13.251637 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.251621    2579 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 17 16:31:13.251708 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.251671    2579 server.go:1295] "Started kubelet"
Apr 17 16:31:13.251802 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.251754    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 17 16:31:13.251895 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.251811    2579 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 17 16:31:13.251895 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.251871    2579 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 17 16:31:13.252471 ip-10-0-141-140 systemd[1]: Started Kubernetes Kubelet.
Apr 17 16:31:13.253650 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.253632    2579 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 17 16:31:13.254388 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.254376    2579 server.go:317] "Adding debug handlers to kubelet server"
Apr 17 16:31:13.259441 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.259390    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 17 16:31:13.260066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260048    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 17 16:31:13.260701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260686    2579 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 17 16:31:13.260701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260688    2579 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 17 16:31:13.260851 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260721    2579 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 17 16:31:13.260851 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260809    2579 reconstruct.go:97] "Volume reconstruction finished"
Apr 17 16:31:13.260851 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.260818    2579 reconciler.go:26] "Reconciler: start to sync state"
Apr 17 16:31:13.261060 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.261041    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.262027 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.262009    2579 factory.go:55] Registering systemd factory
Apr 17 16:31:13.262114 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.262070    2579 factory.go:223] Registration of the systemd container factory successfully
Apr 17 16:31:13.262165 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.262121    2579 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:13.262993 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.262978    2579 factory.go:153] Registering CRI-O factory
Apr 17 16:31:13.263086 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263077    2579 factory.go:223] Registration of the crio container factory successfully
Apr 17 16:31:13.263191 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263028    2579 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-140.ec2.internal" not found
Apr 17 16:31:13.263273 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263263    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 17 16:31:13.263378 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263369    2579 factory.go:103] Registering Raw factory
Apr 17 16:31:13.263449 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263443    2579 manager.go:1196] Started watching for new ooms in manager
Apr 17 16:31:13.263884 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.263867    2579 manager.go:319] Starting recovery of all containers
Apr 17 16:31:13.264455 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.264414    2579 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 17 16:31:13.265189 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.265171    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-140.ec2.internal\" not found" node="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.273832 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.273667    2579 manager.go:324] Recovery completed
Apr 17 16:31:13.276266 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.276247    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Apr 17 16:31:13.278484 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.278468    2579 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-140.ec2.internal" not found
Apr 17 16:31:13.279098 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.279085    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.281775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.281748    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.281844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.281788    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.281844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.281798    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.282240 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.282226    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 17 16:31:13.282240 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.282237    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 17 16:31:13.282339 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.282253    2579 state_mem.go:36] "Initialized new in-memory state store"
Apr 17 16:31:13.284945 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.284933    2579 policy_none.go:49] "None policy: Start"
Apr 17 16:31:13.284981 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.284949    2579 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 17 16:31:13.284981 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.284959    2579 state_mem.go:35] "Initializing new in-memory state store"
Apr 17 16:31:13.315607 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.315592    2579 manager.go:341] "Starting Device Plugin manager"
Apr 17 16:31:13.315697 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.315631    2579 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 17 16:31:13.315697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.315644    2579 server.go:85] "Starting device plugin registration server"
Apr 17 16:31:13.315938 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.315909    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.315929    2579 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.316076    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.316175    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.316184    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.316572    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.316599    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.334036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.333746    2579 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-140.ec2.internal" not found
Apr 17 16:31:13.382697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.382667    2579 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 17 16:31:13.384003 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.383980    2579 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 17 16:31:13.384090 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.384009    2579 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 17 16:31:13.384090 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.384028    2579 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 17 16:31:13.384090 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.384034    2579 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 17 16:31:13.384090 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.384063    2579 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 17 16:31:13.386288 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.386272    2579 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:13.416349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.416321    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.417276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.417262    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.417347 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.417289    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.417347 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.417301    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.417347 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.417325    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.425914 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.425901    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.425956 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.425922    2579 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-140.ec2.internal\": node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.443339 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.443323    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.484622 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.484595    2579 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"]
Apr 17 16:31:13.484702 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.484667    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.485453 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.485436    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.485535 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.485464    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.485535 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.485474    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.486770 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.486750    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.486912 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.486899    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.486959 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.486929    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.487446 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487429    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.487521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487457    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.487521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487432    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.487521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487471    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.487521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487488    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.487521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.487500    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.488706 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.488693    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.488752 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.488718    2579 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 17 16:31:13.489362 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.489339    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientMemory"
Apr 17 16:31:13.489443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.489371    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasNoDiskPressure"
Apr 17 16:31:13.489443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.489384    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeHasSufficientPID"
Apr 17 16:31:13.515299 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.515259    2579 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-140.ec2.internal\" not found" node="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.519463 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.519449    2579 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-140.ec2.internal\" not found" node="ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.543589 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.543572    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.562746 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.562727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.562829 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.562751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.562829 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.562784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/731cb92bc1983e14e5fcbbe29c5e1e5c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-140.ec2.internal\" (UID: \"731cb92bc1983e14e5fcbbe29c5e1e5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.644563 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.644538    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.663212 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.663306 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663158    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.663306 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663248    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.663306 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663268    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/731cb92bc1983e14e5fcbbe29c5e1e5c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-140.ec2.internal\" (UID: \"731cb92bc1983e14e5fcbbe29c5e1e5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.663306 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/731cb92bc1983e14e5fcbbe29c5e1e5c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-140.ec2.internal\" (UID: \"731cb92bc1983e14e5fcbbe29c5e1e5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.663444 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.663314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f65ab440d7a211f96530dfbbb37b622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal\" (UID: \"7f65ab440d7a211f96530dfbbb37b622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.744975 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.744947    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.818623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.818552    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.822289 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:13.822273    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:13.845203 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.845177    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:13.945696 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:13.945653    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:14.046167 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.046130    2579 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-140.ec2.internal\" not found"
Apr 17 16:31:14.061698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.061678    2579 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:14.161213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.161186    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:14.165349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.165334    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 17 16:31:14.165549 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.165531    2579 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 17 16:31:14.165549 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.165539    2579 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 17 16:31:14.165647 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.165542    2579 kubelet.go:3342] "Failed creating a mirror pod" err="Post \"https://abff99fd067734066b555a738e1885e1-da717e0a41fe01e3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.141.140:44618->34.237.188.198:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:14.165647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.165567    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:14.182051 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.182024    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 17 16:31:14.236599 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.236561    2579 apiserver.go:52] "Watching apiserver"
Apr 17 16:31:14.247845 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.247821    2579 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 17 16:31:14.248233 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.248200    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-bsgct","openshift-multus/multus-additional-cni-plugins-25sc2","openshift-multus/network-metrics-daemon-fwn9m","openshift-network-operator/iptables-alerter-jf79v","kube-system/konnectivity-agent-67c86","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x","openshift-dns/node-resolver-j5cpk","openshift-image-registry/node-ca-76q52","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal","openshift-multus/multus-cqjg6","openshift-network-diagnostics/network-check-target-t84cs","openshift-ovn-kubernetes/ovnkube-node-2l4rx"]
Apr 17 16:31:14.250664 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.250646    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.250876 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.250835    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-16 16:26:13 +0000 UTC" deadline="2027-10-15 16:04:17.824868207 +0000 UTC"
Apr 17 16:31:14.250876 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.250867    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13103h33m3.574004556s"
Apr 17 16:31:14.251968 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.251946    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.252891 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.252873    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.252987 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.252874    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.252987 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.252948    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wkvc\""
Apr 17 16:31:14.253246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.253229    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.253320 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.253299    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:14.254175 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254146    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 17 16:31:14.254275 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254255    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 17 16:31:14.254445 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254428    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pvr62\""
Apr 17 16:31:14.254518 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254483    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.254651 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254636    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.254799 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254777    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 17 16:31:14.254877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.254857    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.255679 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.255661    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.257006 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.256984    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.257244 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257225    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.257329 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257315    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 17 16:31:14.257446 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257432    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.257539 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257520    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bsg2m\""
Apr 17 16:31:14.257785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257753    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 17 16:31:14.257868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.257753    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 17 16:31:14.258160 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.258141    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7mm8l\""
Apr 17 16:31:14.258230 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.258186    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.258773 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.258744    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8nqx2\""
Apr 17 16:31:14.258858 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.258819    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 17 16:31:14.259542 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.259518    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.259669 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.259650    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 17 16:31:14.260788 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.260756    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.261025 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.261008    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.262014 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.261995    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.262346 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.262329    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s6sps\""
Apr 17 16:31:14.262422 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.262397    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:14.262501 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.262470    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:14.262566 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.262520    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.262855 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.262823    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 17 16:31:14.263380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.263363    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 17 16:31:14.263854 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.263835    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.263951 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.263915    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.263951 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.263923    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ps24l\""
Apr 17 16:31:14.264047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.263953    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m4s7h\""
Apr 17 16:31:14.264047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.264035    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.265082 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.264635    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.265891 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.265874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-kubernetes\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266003 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.265904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.266003 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.265930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797h6\" (UniqueName: \"kubernetes.io/projected/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-kube-api-access-797h6\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.266117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.265999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.266117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-device-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.266117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4th\" (UniqueName: \"kubernetes.io/projected/9f980360-c063-4b18-8ccb-25637356228f-kube-api-access-dm4th\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-host-slash\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.266117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8ca92f6-eb37-4066-b492-157ff72c4e53-konnectivity-ca\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.266290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-sys\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-modprobe-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-sys-fs\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.266290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648br\" (UniqueName: \"kubernetes.io/projected/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-kube-api-access-648br\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-multus-certs\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38c0cc14-3fcb-4193-8823-9a4e0a06f777-hosts-file\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-multus\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-tuned\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-tmp\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-kubelet\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-conf-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvrn\" (UniqueName: \"kubernetes.io/projected/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-kube-api-access-kwvrn\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-netns\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-run\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84qj\" (UniqueName: \"kubernetes.io/projected/38c0cc14-3fcb-4193-8823-9a4e0a06f777-kube-api-access-q84qj\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-serviceca\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-cni-binary-copy\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.266790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-cnibin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-system-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-lib-modules\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xrj\" (UniqueName: \"kubernetes.io/projected/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-kube-api-access-f7xrj\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtpg\" (UniqueName: \"kubernetes.io/projected/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kube-api-access-6wtpg\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c0cc14-3fcb-4193-8823-9a4e0a06f777-tmp-dir\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-socket-dir-parent\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.266987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-etc-kubernetes\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysconfig\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-conf\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8ca92f6-eb37-4066-b492-157ff72c4e53-agent-certs\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrckr\" (UniqueName: \"kubernetes.io/projected/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-kube-api-access-jrckr\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-registration-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-os-release\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-k8s-cni-cncf-io\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.267452 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-systemd\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-bin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-system-cni-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-host\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cnibin\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-hostroot\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.267869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-multus-daemon-config\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-socket-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-host\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-var-lib-kubelet\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-iptables-alerter-script\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-os-release\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.268380 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.268978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268411    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 17 16:31:14.268978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268531    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jhkwc\""
Apr 17 16:31:14.268978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268559    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 17 16:31:14.268978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268785    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 17 16:31:14.268978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.268898    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 17 16:31:14.270356 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.269413    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 17 16:31:14.270356 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.269599    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 17 16:31:14.276716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.276700    2579 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 17 16:31:14.309815 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.309777    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvjlc"
Apr 17 16:31:14.318118 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.318088    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvjlc"
Apr 17 16:31:14.353972 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.353939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731cb92bc1983e14e5fcbbe29c5e1e5c.slice/crio-2b20d97b9ba688526ef8aab8709603c39ebbb324e026bb99df35e618132bc8d3 WatchSource:0}: Error finding container 2b20d97b9ba688526ef8aab8709603c39ebbb324e026bb99df35e618132bc8d3: Status 404 returned error can't find the container with id 2b20d97b9ba688526ef8aab8709603c39ebbb324e026bb99df35e618132bc8d3
Apr 17 16:31:14.354210 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.354190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f65ab440d7a211f96530dfbbb37b622.slice/crio-fbfc7f856edb0e6407207983faf2cc9e20a8f645e68648d0eafafe9c931b964c WatchSource:0}: Error finding container fbfc7f856edb0e6407207983faf2cc9e20a8f645e68648d0eafafe9c931b964c: Status 404 returned error can't find the container with id fbfc7f856edb0e6407207983faf2cc9e20a8f645e68648d0eafafe9c931b964c
Apr 17 16:31:14.360162 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.360149    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 16:31:14.362252 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.362234    2579 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 17 16:31:14.368508 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368490    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysconfig\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.368606 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368521    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-conf\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.368606 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368545    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8ca92f6-eb37-4066-b492-157ff72c4e53-agent-certs\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.368606 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368572    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrckr\" (UniqueName: \"kubernetes.io/projected/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-kube-api-access-jrckr\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368617    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysconfig\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368612    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-registration-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368666    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-conf\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368692    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-os-release\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368704    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-registration-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.368751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368719    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-k8s-cni-cncf-io\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368775    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-k8s-cni-cncf-io\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368786    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-systemd\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-systemd\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-ovn\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-os-release\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-env-overrides\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368868    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-bin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368889    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-system-cni-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368907    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368927    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-host\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-bin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-systemd-units\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368904    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-node-log\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.368942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-system-cni-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-host\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-script-lib\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369049    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cnibin\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369080    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-hostroot\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369085    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cnibin\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-hostroot\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369119    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-multus-daemon-config\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369154    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-socket-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369179    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-host\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369202    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-var-lib-kubelet\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369226    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-iptables-alerter-script\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-kubelet\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369264    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-socket-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-etc-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369300    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-var-lib-kubelet\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369305    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-os-release\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-host\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.369822 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369341    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369371    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-os-release\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369406    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-kubernetes\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369438    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-kubernetes\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369461    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-797h6\" (UniqueName: \"kubernetes.io/projected/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-kube-api-access-797h6\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-slash\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-var-lib-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8f806c8-de16-4bb6-bb39-1871692083fc-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.369533    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369550    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369571    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-device-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.369614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:14.869567991 +0000 UTC m=+2.110226410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369647    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4th\" (UniqueName: \"kubernetes.io/projected/9f980360-c063-4b18-8ccb-25637356228f-kube-api-access-dm4th\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369673    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-host-slash\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369698    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8ca92f6-eb37-4066-b492-157ff72c4e53-konnectivity-ca\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369721    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-sys\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.370698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369743    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-modprobe-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-systemd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8mf\" (UniqueName: \"kubernetes.io/projected/a8f806c8-de16-4bb6-bb39-1871692083fc-kube-api-access-8c8mf\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-device-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-multus-daemon-config\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-iptables-alerter-script\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369925    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-sys\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369959    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-sys-fs\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369989    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-648br\" (UniqueName: \"kubernetes.io/projected/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-kube-api-access-648br\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.369995    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-host-slash\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370021    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-multus-certs\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370073    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-sys-fs\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-multus-certs\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370088    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-modprobe-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-netns\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.371544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370192    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370203    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370226    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38c0cc14-3fcb-4193-8823-9a4e0a06f777-hosts-file\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370255    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370293    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-multus\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370319    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-tuned\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370344    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-tmp\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370347    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8ca92f6-eb37-4066-b492-157ff72c4e53-konnectivity-ca\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370364    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38c0cc14-3fcb-4193-8823-9a4e0a06f777-hosts-file\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-netd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370449    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-cni-multus\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370471    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-kubelet\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-var-lib-kubelet\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370531    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-conf-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370559    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370584    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvrn\" (UniqueName: \"kubernetes.io/projected/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-kube-api-access-kwvrn\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-log-socket\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-conf-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370638    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-netns\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370663    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-run\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370688    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370712    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q84qj\" (UniqueName: \"kubernetes.io/projected/38c0cc14-3fcb-4193-8823-9a4e0a06f777-kube-api-access-q84qj\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370735    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-serviceca\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370779    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-cni-binary-copy\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-config\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-host-run-netns\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370890    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-cnibin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370909    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-sysctl-d\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-cnibin\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.370998    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-system-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371141    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-lib-modules\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.372908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371169    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xrj\" (UniqueName: \"kubernetes.io/projected/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-kube-api-access-f7xrj\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-bin\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371253    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371281    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtpg\" (UniqueName: \"kubernetes.io/projected/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kube-api-access-6wtpg\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371308    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c0cc14-3fcb-4193-8823-9a4e0a06f777-tmp-dir\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371365    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-socket-dir-parent\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371392    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-etc-kubernetes\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371466    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-etc-kubernetes\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371669    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-serviceca\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c0cc14-3fcb-4193-8823-9a4e0a06f777-tmp-dir\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.371956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-system-cni-dir\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372040    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-lib-modules\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372043    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51b46c0a-a153-4d43-8ea6-e04afe2be10a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-run\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f980360-c063-4b18-8ccb-25637356228f-cni-binary-copy\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372219    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f980360-c063-4b18-8ccb-25637356228f-multus-socket-dir-parent\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372729    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-etc-tuned\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.373472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.372888    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8ca92f6-eb37-4066-b492-157ff72c4e53-agent-certs\") pod \"konnectivity-agent-67c86\" (UID: \"b8ca92f6-eb37-4066-b492-157ff72c4e53\") " pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.374035 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.373362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-tmp\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.377116 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.377084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrckr\" (UniqueName: \"kubernetes.io/projected/0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab-kube-api-access-jrckr\") pod \"multus-additional-cni-plugins-25sc2\" (UID: \"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab\") " pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.377901 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.377876    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797h6\" (UniqueName: \"kubernetes.io/projected/17fd98b1-2f78-4fb1-aee8-a9333fd592b4-kube-api-access-797h6\") pod \"iptables-alerter-jf79v\" (UID: \"17fd98b1-2f78-4fb1-aee8-a9333fd592b4\") " pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.378717 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.378620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-648br\" (UniqueName: \"kubernetes.io/projected/4bd685f1-9eee-4a49-9473-4db1d50bd8b8-kube-api-access-648br\") pod \"node-ca-76q52\" (UID: \"4bd685f1-9eee-4a49-9473-4db1d50bd8b8\") " pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.378920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.378902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4th\" (UniqueName: \"kubernetes.io/projected/9f980360-c063-4b18-8ccb-25637356228f-kube-api-access-dm4th\") pod \"multus-cqjg6\" (UID: \"9f980360-c063-4b18-8ccb-25637356228f\") " pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.381838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.381821    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cqjg6"
Apr 17 16:31:14.383007 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.382948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtpg\" (UniqueName: \"kubernetes.io/projected/51b46c0a-a153-4d43-8ea6-e04afe2be10a-kube-api-access-6wtpg\") pod \"aws-ebs-csi-driver-node-8vk6x\" (UID: \"51b46c0a-a153-4d43-8ea6-e04afe2be10a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.383448 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.383416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvrn\" (UniqueName: \"kubernetes.io/projected/a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611-kube-api-access-kwvrn\") pod \"tuned-bsgct\" (UID: \"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611\") " pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.383774 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.383737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xrj\" (UniqueName: \"kubernetes.io/projected/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-kube-api-access-f7xrj\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.383846 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.383831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84qj\" (UniqueName: \"kubernetes.io/projected/38c0cc14-3fcb-4193-8823-9a4e0a06f777-kube-api-access-q84qj\") pod \"node-resolver-j5cpk\" (UID: \"38c0cc14-3fcb-4193-8823-9a4e0a06f777\") " pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.386929 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.386890    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal" event={"ID":"7f65ab440d7a211f96530dfbbb37b622","Type":"ContainerStarted","Data":"fbfc7f856edb0e6407207983faf2cc9e20a8f645e68648d0eafafe9c931b964c"}
Apr 17 16:31:14.387914 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.387890    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal" event={"ID":"731cb92bc1983e14e5fcbbe29c5e1e5c","Type":"ContainerStarted","Data":"2b20d97b9ba688526ef8aab8709603c39ebbb324e026bb99df35e618132bc8d3"}
Apr 17 16:31:14.389550 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.389532    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f980360_c063_4b18_8ccb_25637356228f.slice/crio-3f84866f80dc77e98c1b24f9314740126d9e6601b83d7fa42349263f5fee34e8 WatchSource:0}: Error finding container 3f84866f80dc77e98c1b24f9314740126d9e6601b83d7fa42349263f5fee34e8: Status 404 returned error can't find the container with id 3f84866f80dc77e98c1b24f9314740126d9e6601b83d7fa42349263f5fee34e8
Apr 17 16:31:14.472174 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472099    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-log-socket\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472174 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472157    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-config\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472176    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-bin\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472195    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-ovn\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472210    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-env-overrides\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472237    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-systemd-units\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-log-socket\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472260    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-node-log\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472272    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-bin\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472285    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472320    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-script-lib\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-node-log\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472376 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472370    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-kubelet\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472393    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-etc-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-etc-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472457    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-slash\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472491    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-var-lib-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472497    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-systemd-units\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472504    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-slash\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472517    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8f806c8-de16-4bb6-bb39-1871692083fc-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-kubelet\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472519    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-ovn\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-var-lib-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472581    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-systemd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472596    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8mf\" (UniqueName: \"kubernetes.io/projected/a8f806c8-de16-4bb6-bb39-1871692083fc-kube-api-access-8c8mf\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472618    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472643    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-systemd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472660    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-netns\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.472886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472711    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472740    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472789    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-netd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472845    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-config\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472858    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-cni-netd\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-env-overrides\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-host-run-netns\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.472902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8f806c8-de16-4bb6-bb39-1871692083fc-run-openvswitch\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.473384 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.473004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8f806c8-de16-4bb6-bb39-1871692083fc-ovnkube-script-lib\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.474827 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.474812    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8f806c8-de16-4bb6-bb39-1871692083fc-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.478663 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.478649    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:14.478706 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.478666    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:14.478706 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.478676    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:14.478772 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.478731    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:14.978718184 +0000 UTC m=+2.219376615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:14.481444 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.481420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8mf\" (UniqueName: \"kubernetes.io/projected/a8f806c8-de16-4bb6-bb39-1871692083fc-kube-api-access-8c8mf\") pod \"ovnkube-node-2l4rx\" (UID: \"a8f806c8-de16-4bb6-bb39-1871692083fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.584393 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.584365    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsgct"
Apr 17 16:31:14.591058 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.591032    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ec9e8b_2f1a_4c9c_96a5_23d9e8deb611.slice/crio-11c25ce7650319569e0e95072402c719076fe29ad307073b81dba71ea4e33676 WatchSource:0}: Error finding container 11c25ce7650319569e0e95072402c719076fe29ad307073b81dba71ea4e33676: Status 404 returned error can't find the container with id 11c25ce7650319569e0e95072402c719076fe29ad307073b81dba71ea4e33676
Apr 17 16:31:14.596918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.596901    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25sc2"
Apr 17 16:31:14.602056 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.602029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1bae00_7e41_4dd8_85a3_04bf2d90a6ab.slice/crio-ed23f8a718dd4a45ded7735b10d1e04a165dcea1fdee3479f52688b755951a20 WatchSource:0}: Error finding container ed23f8a718dd4a45ded7735b10d1e04a165dcea1fdee3479f52688b755951a20: Status 404 returned error can't find the container with id ed23f8a718dd4a45ded7735b10d1e04a165dcea1fdee3479f52688b755951a20
Apr 17 16:31:14.612623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.612607    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jf79v"
Apr 17 16:31:14.618552 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.618527    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fd98b1_2f78_4fb1_aee8_a9333fd592b4.slice/crio-b0164d4c8df8e258be88240ae920a7e1084716f7e49b1bd1039269c4717d484e WatchSource:0}: Error finding container b0164d4c8df8e258be88240ae920a7e1084716f7e49b1bd1039269c4717d484e: Status 404 returned error can't find the container with id b0164d4c8df8e258be88240ae920a7e1084716f7e49b1bd1039269c4717d484e
Apr 17 16:31:14.631907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.631891    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:14.656867 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.656848    2579 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:14.659748 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.659725    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x"
Apr 17 16:31:14.665390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.665373    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j5cpk"
Apr 17 16:31:14.665456 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.665369    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b46c0a_a153_4d43_8ea6_e04afe2be10a.slice/crio-6e930a1a7db2d0489073eaa719fadafbe4980c7603f874ece14bf658b08f9634 WatchSource:0}: Error finding container 6e930a1a7db2d0489073eaa719fadafbe4980c7603f874ece14bf658b08f9634: Status 404 returned error can't find the container with id 6e930a1a7db2d0489073eaa719fadafbe4980c7603f874ece14bf658b08f9634
Apr 17 16:31:14.670844 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.670826    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c0cc14_3fcb_4193_8823_9a4e0a06f777.slice/crio-696a1f60a033f996cebb7f478a1de473b909395baf74d28dd8ac6b566197b224 WatchSource:0}: Error finding container 696a1f60a033f996cebb7f478a1de473b909395baf74d28dd8ac6b566197b224: Status 404 returned error can't find the container with id 696a1f60a033f996cebb7f478a1de473b909395baf74d28dd8ac6b566197b224
Apr 17 16:31:14.671992 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.671951    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76q52"
Apr 17 16:31:14.677918 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.677899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd685f1_9eee_4a49_9473_4db1d50bd8b8.slice/crio-d5c69ccb4316d182e59bbd3272c771ec9088b586d8f2be0b71570a16804f7599 WatchSource:0}: Error finding container d5c69ccb4316d182e59bbd3272c771ec9088b586d8f2be0b71570a16804f7599: Status 404 returned error can't find the container with id d5c69ccb4316d182e59bbd3272c771ec9088b586d8f2be0b71570a16804f7599
Apr 17 16:31:14.688579 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.688564    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:14.694025 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:14.694005    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f806c8_de16_4bb6_bb39_1871692083fc.slice/crio-46f75cf27bf9c6187ae4b1e5923a40d68bec6615da6eaf0568330403b4e4a3c4 WatchSource:0}: Error finding container 46f75cf27bf9c6187ae4b1e5923a40d68bec6615da6eaf0568330403b4e4a3c4: Status 404 returned error can't find the container with id 46f75cf27bf9c6187ae4b1e5923a40d68bec6615da6eaf0568330403b4e4a3c4
Apr 17 16:31:14.875750 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:14.875717    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:14.875918 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.875878    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:14.875973 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:14.875936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:15.875919786 +0000 UTC m=+3.116578200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:15.076857 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.076798    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:15.077040 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.077004    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:15.077040 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.077023    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:15.077040 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.077035    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:15.077206 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.077089    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:16.077071365 +0000 UTC m=+3.317729801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:15.319701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.319528    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-16 16:26:14 +0000 UTC" deadline="2027-12-07 14:35:56.141768041 +0000 UTC"
Apr 17 16:31:15.319701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.319564    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14374h4m40.822207922s"
Apr 17 16:31:15.367856 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.367828    2579 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:15.406303 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.406265    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jf79v" event={"ID":"17fd98b1-2f78-4fb1-aee8-a9333fd592b4","Type":"ContainerStarted","Data":"b0164d4c8df8e258be88240ae920a7e1084716f7e49b1bd1039269c4717d484e"}
Apr 17 16:31:15.420951 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.420896    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerStarted","Data":"ed23f8a718dd4a45ded7735b10d1e04a165dcea1fdee3479f52688b755951a20"}
Apr 17 16:31:15.432382 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.432351    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsgct" event={"ID":"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611","Type":"ContainerStarted","Data":"11c25ce7650319569e0e95072402c719076fe29ad307073b81dba71ea4e33676"}
Apr 17 16:31:15.456391 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.456357    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cqjg6" event={"ID":"9f980360-c063-4b18-8ccb-25637356228f","Type":"ContainerStarted","Data":"3f84866f80dc77e98c1b24f9314740126d9e6601b83d7fa42349263f5fee34e8"}
Apr 17 16:31:15.470142 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.470109    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"46f75cf27bf9c6187ae4b1e5923a40d68bec6615da6eaf0568330403b4e4a3c4"}
Apr 17 16:31:15.490992 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.490960    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j5cpk" event={"ID":"38c0cc14-3fcb-4193-8823-9a4e0a06f777","Type":"ContainerStarted","Data":"696a1f60a033f996cebb7f478a1de473b909395baf74d28dd8ac6b566197b224"}
Apr 17 16:31:15.503316 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.503271    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x" event={"ID":"51b46c0a-a153-4d43-8ea6-e04afe2be10a","Type":"ContainerStarted","Data":"6e930a1a7db2d0489073eaa719fadafbe4980c7603f874ece14bf658b08f9634"}
Apr 17 16:31:15.505751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.505723    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67c86" event={"ID":"b8ca92f6-eb37-4066-b492-157ff72c4e53","Type":"ContainerStarted","Data":"5ff05db5eb113af2519a8a5fcc4895a50f9e0d5c831df0082ee58aeefe39007d"}
Apr 17 16:31:15.514453 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.514384    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76q52" event={"ID":"4bd685f1-9eee-4a49-9473-4db1d50bd8b8","Type":"ContainerStarted","Data":"d5c69ccb4316d182e59bbd3272c771ec9088b586d8f2be0b71570a16804f7599"}
Apr 17 16:31:15.697701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.697361    2579 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 17 16:31:15.884326 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:15.883578    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:15.884326 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.883753    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:15.884326 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:15.883831    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:17.883812271 +0000 UTC m=+5.124470690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:16.085210 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:16.085126    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:16.085452 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.085313    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:16.085452 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.085348    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:16.085452 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.085363    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:16.085452 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.085422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:18.085404022 +0000 UTC m=+5.326062438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:16.320637 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:16.320592    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-16 16:26:14 +0000 UTC" deadline="2027-10-07 19:48:02.918362643 +0000 UTC"
Apr 17 16:31:16.320637 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:16.320639    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12915h16m46.597728292s"
Apr 17 16:31:16.384642 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:16.384609    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:16.384833 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.384747    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:16.385211 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:16.385189    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:16.385299 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:16.385284    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:17.905297 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:17.905252    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:17.905855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:17.905425    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:17.905855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:17.905497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:21.905477091 +0000 UTC m=+9.146135522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:18.107632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:18.107515    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:18.107836 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.107663    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:18.107836 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.107683    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:18.107836 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.107695    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:18.107836 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.107752    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:22.107735096 +0000 UTC m=+9.348393509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:18.384882 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:18.384306    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:18.384882 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.384435    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:18.384882 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:18.384509    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:18.384882 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:18.384627    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:20.384838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:20.384246    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:20.384838 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:20.384389    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:20.384838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:20.384246    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:20.384838 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:20.384830    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:21.936512 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:21.936420    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:21.937001 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:21.936545    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:21.937001 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:21.936617    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:29.936595562 +0000 UTC m=+17.177253991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:22.137840 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:22.137727    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:22.138078 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.137898    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:22.138078 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.137925    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:22.138078 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.137938    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:22.138078 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.138004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:30.137985178 +0000 UTC m=+17.378643612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:22.384313 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:22.384274    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:22.384506 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.384421    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:22.384506 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:22.384475    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:22.384657 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:22.384610    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:23.363571 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.363023    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fgtpp"]
Apr 17 16:31:23.369358 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.369062    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.369358 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:23.369142    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:23.447900 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.447865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-dbus\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.448081 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.447949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-kubelet-config\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.448081 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.447977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.548879 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.548843    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-kubelet-config\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.549041 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.548887    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.549041 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.548942    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-dbus\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.549041 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.548991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-kubelet-config\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.549185 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:23.549102    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:23.549185 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:23.549112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-dbus\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:23.549185 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:23.549166    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:24.049146032 +0000 UTC m=+11.289804466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:24.052866 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:24.052828    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:24.053039 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:24.052973    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:24.053084 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:24.053073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:25.05304934 +0000 UTC m=+12.293707771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:24.384416 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:24.384387    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:24.384746 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:24.384449    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:24.384746 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:24.384557    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:24.384746 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:24.384665    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:25.061112 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:25.061073    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:25.061287 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:25.061263    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:25.061349 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:25.061337    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:27.061317806 +0000 UTC m=+14.301976224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:25.384538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:25.384508    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:25.384967 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:25.384608    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:26.384251 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:26.384212    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:26.384432 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:26.384218    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:26.384432 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:26.384334    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:26.384514 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:26.384424    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:27.078318 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:27.078279    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:27.078805 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:27.078392    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:27.078805 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:27.078452    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:31.078433681 +0000 UTC m=+18.319092106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:27.384284 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:27.384252    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:27.384466 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:27.384384    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:28.385203 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:28.385169    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:28.385785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:28.385248    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:28.385785 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:28.385385    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:28.385785 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:28.385519    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:29.384585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:29.384553    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:29.384799 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:29.384666    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:29.999342 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:29.999313    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:29.999798 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:29.999471    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:29.999798 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:29.999532    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:45.999512378 +0000 UTC m=+33.240170809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 17 16:31:30.200690 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:30.200656    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:30.200901 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.200874    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 17 16:31:30.200901 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.200900    2579 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 17 16:31:30.201008 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.200913    2579 projected.go:194] Error preparing data for projected volume kube-api-access-d8r26 for pod openshift-network-diagnostics/network-check-target-t84cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:30.201008 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.200977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26 podName:e9dc640a-deef-48d0-8af4-5279f7023a59 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:46.200958668 +0000 UTC m=+33.441617089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8r26" (UniqueName: "kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26") pod "network-check-target-t84cs" (UID: "e9dc640a-deef-48d0-8af4-5279f7023a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 17 16:31:30.385278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:30.385246    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:30.385460 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.385371    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:30.385460 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:30.385390    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:30.385578 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:30.385515    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:31.108553 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:31.108519    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:31.109002 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:31.108676    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:31.109002 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:31.108746    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:39.108730515 +0000 UTC m=+26.349388947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:31.384290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:31.384262    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:31.384461 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:31.384371    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:32.384802 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.384741    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:32.385168 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:32.384893    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:32.385272 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.385258    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:32.385380 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:32.385359    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:32.556125 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.555950    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal" event={"ID":"731cb92bc1983e14e5fcbbe29c5e1e5c","Type":"ContainerStarted","Data":"835d4753b40e82167041cf8fe99f8be10b1a84d05361d5e63e9f788820594177"}
Apr 17 16:31:32.556243 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.556148    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:32.557558 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.557535    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsgct" event={"ID":"a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611","Type":"ContainerStarted","Data":"7fe17db427c1dbe4dbd6ba2c34c51911775c5c1de90b0fcca46fbe30324c7998"}
Apr 17 16:31:32.559196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.559172    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cqjg6" event={"ID":"9f980360-c063-4b18-8ccb-25637356228f","Type":"ContainerStarted","Data":"738b8030b5c46f95f431b3f7741fd83dc6a12d7aea34f8c5de262d416b7d679d"}
Apr 17 16:31:32.561387 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.561283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:31:32.561731 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.561696    2579 generic.go:358] "Generic (PLEG): container finished" podID="a8f806c8-de16-4bb6-bb39-1871692083fc" containerID="0d0da3ca3d71ff821fb329adcf548abace99d42531ec9fa40a0dd57892bdc67a" exitCode=1
Apr 17 16:31:32.561830 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.561734    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerDied","Data":"0d0da3ca3d71ff821fb329adcf548abace99d42531ec9fa40a0dd57892bdc67a"}
Apr 17 16:31:32.561830 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.561754    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"9d462b5d278946b8a25e60992a5f893fc47b28e31ff24ff02b927b67c6324736"}
Apr 17 16:31:32.565886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.565864    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 17 16:31:32.566324 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.566305    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"]
Apr 17 16:31:32.574457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.573218    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bsgct" podStartSLOduration=1.978198015 podStartE2EDuration="19.57320324s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.592546605 +0000 UTC m=+1.833205019" lastFinishedPulling="2026-04-17 16:31:32.187551817 +0000 UTC m=+19.428210244" observedRunningTime="2026-04-17 16:31:32.572800072 +0000 UTC m=+19.813458509" watchObservedRunningTime="2026-04-17 16:31:32.57320324 +0000 UTC m=+19.813861689"
Apr 17 16:31:32.585788 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.585730    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal" podStartSLOduration=0.585716019 podStartE2EDuration="585.716019ms" podCreationTimestamp="2026-04-17 16:31:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:31:32.585452801 +0000 UTC m=+19.826111238" watchObservedRunningTime="2026-04-17 16:31:32.585716019 +0000 UTC m=+19.826374458"
Apr 17 16:31:32.599948 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:32.599906    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-cqjg6" podStartSLOduration=1.7986561129999998 podStartE2EDuration="19.599888511s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.391035831 +0000 UTC m=+1.631694248" lastFinishedPulling="2026-04-17 16:31:32.192268229 +0000 UTC m=+19.432926646" observedRunningTime="2026-04-17 16:31:32.599590884 +0000 UTC m=+19.840249320" watchObservedRunningTime="2026-04-17 16:31:32.599888511 +0000 UTC m=+19.840546949"
Apr 17 16:31:33.387234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.387069    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:33.387787 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:33.387296    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:33.564776 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.564725    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x" event={"ID":"51b46c0a-a153-4d43-8ea6-e04afe2be10a","Type":"ContainerStarted","Data":"24676c633f11ff642527d0b2c85860e5f9a3690d92d3035ee8fd5501a94f738d"}
Apr 17 16:31:33.566264 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.566093    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67c86" event={"ID":"b8ca92f6-eb37-4066-b492-157ff72c4e53","Type":"ContainerStarted","Data":"d4b87589ec04c8557952e4cf774158577a28178b9478362733cb177c28089a93"}
Apr 17 16:31:33.568025 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.567997    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76q52" event={"ID":"4bd685f1-9eee-4a49-9473-4db1d50bd8b8","Type":"ContainerStarted","Data":"1fe4de1c08a4a1961483ac309f6d7dca5d0e87df2ae499358b00a2ecce8c7894"}
Apr 17 16:31:33.569670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.569642    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jf79v" event={"ID":"17fd98b1-2f78-4fb1-aee8-a9333fd592b4","Type":"ContainerStarted","Data":"50b54834d9630bd2d44294abaf0d3023d2f844c54f073aa72cab8c5491e57c63"}
Apr 17 16:31:33.571125 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.571098    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="4f520ac5fe51a67b38d846b964d421a111b2fd2b817f7e0eaf13cdefbae83f86" exitCode=0
Apr 17 16:31:33.571214 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.571176    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"4f520ac5fe51a67b38d846b964d421a111b2fd2b817f7e0eaf13cdefbae83f86"}
Apr 17 16:31:33.573118 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.573095    2579 generic.go:358] "Generic (PLEG): container finished" podID="7f65ab440d7a211f96530dfbbb37b622" containerID="1e28724afac58cddddb93521be3af69dcbc8cf35b13969ac463bc22b0ee1eeff" exitCode=0
Apr 17 16:31:33.573213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.573179    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal" event={"ID":"7f65ab440d7a211f96530dfbbb37b622","Type":"ContainerDied","Data":"1e28724afac58cddddb93521be3af69dcbc8cf35b13969ac463bc22b0ee1eeff"}
Apr 17 16:31:33.576068 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.576049    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:31:33.576453 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.576405    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"6f9827d213e583ec6cb2f9237c3f5194833265e307a72b1f3c2b3d0e14c0c851"}
Apr 17 16:31:33.576453 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.576431    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"bc547be7d75fb915d955b8394ef361590a4cc22a4eaf6e05a3d60f2d53f45e27"}
Apr 17 16:31:33.576453 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.576447    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"7abd992596be25893121ce7e5875df9b124fef55bb35ff146d07e0d9ea31267a"}
Apr 17 16:31:33.576630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.576459    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"4ba7ea32109dccdffc0d34690e48d08a76b9709b3fb1aa18c83adcaf84b62822"}
Apr 17 16:31:33.577943 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.577921    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j5cpk" event={"ID":"38c0cc14-3fcb-4193-8823-9a4e0a06f777","Type":"ContainerStarted","Data":"04014c5bae67fd8fc71f379b7847b0ae062488efd3321c48dae7abc1bd39c660"}
Apr 17 16:31:33.578455 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.578438    2579 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:33.581906 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.581874    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-67c86" podStartSLOduration=3.0658735679999998 podStartE2EDuration="20.581864656s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.640086078 +0000 UTC m=+1.880744499" lastFinishedPulling="2026-04-17 16:31:32.15607717 +0000 UTC m=+19.396735587" observedRunningTime="2026-04-17 16:31:33.58186372 +0000 UTC m=+20.822522163" watchObservedRunningTime="2026-04-17 16:31:33.581864656 +0000 UTC m=+20.822523092"
Apr 17 16:31:33.588278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.587842    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 17 16:31:33.588278 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:33.587918    2579 kubelet.go:3342] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-140.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-140.ec2.internal"
Apr 17 16:31:33.626029 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.625976    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-j5cpk" podStartSLOduration=3.14175966 podStartE2EDuration="20.625958659s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.67210151 +0000 UTC m=+1.912759925" lastFinishedPulling="2026-04-17 16:31:32.156300496 +0000 UTC m=+19.396958924" observedRunningTime="2026-04-17 16:31:33.625843349 +0000 UTC m=+20.866501785" watchObservedRunningTime="2026-04-17 16:31:33.625958659 +0000 UTC m=+20.866617098"
Apr 17 16:31:33.638942 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.638896    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-76q52" podStartSLOduration=3.161999914 podStartE2EDuration="20.63888284s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.679292602 +0000 UTC m=+1.919951020" lastFinishedPulling="2026-04-17 16:31:32.156175521 +0000 UTC m=+19.396833946" observedRunningTime="2026-04-17 16:31:33.638852741 +0000 UTC m=+20.879511177" watchObservedRunningTime="2026-04-17 16:31:33.63888284 +0000 UTC m=+20.879541276"
Apr 17 16:31:33.669889 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:33.669846    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jf79v" podStartSLOduration=3.104754276 podStartE2EDuration="20.669833699s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.619829402 +0000 UTC m=+1.860487815" lastFinishedPulling="2026-04-17 16:31:32.184908812 +0000 UTC m=+19.425567238" observedRunningTime="2026-04-17 16:31:33.669655816 +0000 UTC m=+20.910314251" watchObservedRunningTime="2026-04-17 16:31:33.669833699 +0000 UTC m=+20.910492135"
Apr 17 16:31:34.023390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.023366    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 17 16:31:34.326886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.326672    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-17T16:31:34.023383147Z","UUID":"5f3e3094-50ec-407f-b24c-c5e75e3ce9ca","Handler":null,"Name":"","Endpoint":""}
Apr 17 16:31:34.328855 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.328827    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 17 16:31:34.328855 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.328858    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 17 16:31:34.384817 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.384788    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:34.384974 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.384788    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:34.384974 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:34.384892    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:34.384974 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:34.384962    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:34.582532 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.582455    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal" event={"ID":"7f65ab440d7a211f96530dfbbb37b622","Type":"ContainerStarted","Data":"cc90e463a5efe6a9ec40344ad162824ddf90bb4a7b5887640c04e6638b7871b4"}
Apr 17 16:31:34.584992 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:34.584873    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x" event={"ID":"51b46c0a-a153-4d43-8ea6-e04afe2be10a","Type":"ContainerStarted","Data":"9f1042ef4558c3dc0a0103a788300ce383ab66090a9f98418f5617eeb3df3996"}
Apr 17 16:31:35.384540 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.384503    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:35.384843 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:35.384647    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:35.590046 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.590012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:31:35.590420 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.590397    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"acdda0c085692ce8bc8cbc02e36e7ef9fac1570076acb617632f85c15db3c041"}
Apr 17 16:31:35.592360 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.592325    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x" event={"ID":"51b46c0a-a153-4d43-8ea6-e04afe2be10a","Type":"ContainerStarted","Data":"d0dccfc54b8b4e6b9ce4da280e25b2cbb653d9e89febbbf95aea65c40d09ba78"}
Apr 17 16:31:35.612178 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.612137    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vk6x" podStartSLOduration=2.420729503 podStartE2EDuration="22.612123137s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.668081359 +0000 UTC m=+1.908739772" lastFinishedPulling="2026-04-17 16:31:34.859474987 +0000 UTC m=+22.100133406" observedRunningTime="2026-04-17 16:31:35.61177516 +0000 UTC m=+22.852433592" watchObservedRunningTime="2026-04-17 16:31:35.612123137 +0000 UTC m=+22.852781573"
Apr 17 16:31:35.612324 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:35.612307    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-140.ec2.internal" podStartSLOduration=21.612302775 podStartE2EDuration="21.612302775s" podCreationTimestamp="2026-04-17 16:31:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:31:34.597016818 +0000 UTC m=+21.837675253" watchObservedRunningTime="2026-04-17 16:31:35.612302775 +0000 UTC m=+22.852961210"
Apr 17 16:31:36.384213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:36.384175    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:36.384213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:36.384199    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:36.384401 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:36.384316    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:36.384461 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:36.384440    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:37.384930 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.384897    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:37.385485 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:37.385025    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:37.525783 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.525745    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:37.526397 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.526382    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:37.600152 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.599967    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:31:37.600450 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.600428    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"f3c7632685e072cad055fa0b2e9e645d63889670148dbafece2f6e636d476321"}
Apr 17 16:31:37.600964 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.600805    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:37.600964 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.600834    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:37.600964 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.600849    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:37.600964 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.600895    2579 scope.go:117] "RemoveContainer" containerID="0d0da3ca3d71ff821fb329adcf548abace99d42531ec9fa40a0dd57892bdc67a"
Apr 17 16:31:37.601568 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.601399    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-67c86"
Apr 17 16:31:37.620484 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:37.620165    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:38.385209 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.385014    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:38.385888 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.385014    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:38.385888 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:38.385303    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:38.385888 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:38.385343    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:38.603472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.603439    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="27f4d58fca5cefd78f9278fb4bc56a4e9b2cebe26cb1a72af996daac391af740" exitCode=0
Apr 17 16:31:38.603607 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.603523    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"27f4d58fca5cefd78f9278fb4bc56a4e9b2cebe26cb1a72af996daac391af740"}
Apr 17 16:31:38.606522 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.606505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:31:38.606849 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.606828    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" event={"ID":"a8f806c8-de16-4bb6-bb39-1871692083fc","Type":"ContainerStarted","Data":"3de718f6b9e9baa5e771b65fd605385070f512b38896d0b9fb86f91575ecf5ac"}
Apr 17 16:31:38.607156 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.607129    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:38.621301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.621283    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:31:38.652164 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:38.652094    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx" podStartSLOduration=7.98588123 podStartE2EDuration="25.652083738s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.69564332 +0000 UTC m=+1.936301738" lastFinishedPulling="2026-04-17 16:31:32.361845832 +0000 UTC m=+19.602504246" observedRunningTime="2026-04-17 16:31:38.651883247 +0000 UTC m=+25.892541683" watchObservedRunningTime="2026-04-17 16:31:38.652083738 +0000 UTC m=+25.892742533"
Apr 17 16:31:39.168661 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.168628    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:39.168894 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.168809    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:39.168894 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.168888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret podName:95e5b703-00aa-4d7f-b79a-e7bd129e8ce7 nodeName:}" failed. No retries permitted until 2026-04-17 16:31:55.168868174 +0000 UTC m=+42.409526607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret") pod "global-pull-secret-syncer-fgtpp" (UID: "95e5b703-00aa-4d7f-b79a-e7bd129e8ce7") : object "kube-system"/"original-pull-secret" not registered
Apr 17 16:31:39.384494 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.384469    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:39.384626 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.384593    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:39.518033 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.517959    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwn9m"]
Apr 17 16:31:39.518352 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.518104    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:39.518352 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.518222    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:39.523461 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.523437    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fgtpp"]
Apr 17 16:31:39.530866 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.530845    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t84cs"]
Apr 17 16:31:39.530962 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.530950    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:39.531087 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.531066    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:39.610643 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.610617    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="9c9c29316976d21be1ffc8c58f9ddf2e4a814a68ff40ebddffc5466c8f51ec25" exitCode=0
Apr 17 16:31:39.610792 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.610726    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"9c9c29316976d21be1ffc8c58f9ddf2e4a814a68ff40ebddffc5466c8f51ec25"}
Apr 17 16:31:39.612005 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:39.611311    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:39.612005 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:39.611418    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:40.614918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:40.614726    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="c9cfeaebe21721833535ee430912283f516598538a0830be09c328bc5882270d" exitCode=0
Apr 17 16:31:40.614918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:40.614818    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"c9cfeaebe21721833535ee430912283f516598538a0830be09c328bc5882270d"}
Apr 17 16:31:41.384511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:41.384340    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:41.384511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:41.384362    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:41.384511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:41.384339    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:41.384511 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:41.384490    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:41.384844 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:41.384609    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:41.384844 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:41.384688    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:43.385724 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:43.385699    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:43.386228 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:43.385806    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:43.386228 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:43.385846    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:31:43.386228 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:43.385884    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t84cs" podUID="e9dc640a-deef-48d0-8af4-5279f7023a59"
Apr 17 16:31:43.386228 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:43.385923    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:43.386228 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:43.385992    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:31:45.086863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.086835    2579 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-140.ec2.internal" event="NodeReady"
Apr 17 16:31:45.087429 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.086995    2579 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 17 16:31:45.136629 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.136595    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f8sph"]
Apr 17 16:31:45.169867 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.169833    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlp4"]
Apr 17 16:31:45.170007 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.169984    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.173213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.173025    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 17 16:31:45.173213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.173081    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f96dz\""
Apr 17 16:31:45.173382 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.173282    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 17 16:31:45.185620 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.185600    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f8sph"]
Apr 17 16:31:45.185620 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.185624    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlp4"]
Apr 17 16:31:45.185780 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.185720    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.188426 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.188405    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 17 16:31:45.188993 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.188972    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 17 16:31:45.189135 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.189088    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 17 16:31:45.189269 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.189249    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txqpq\""
Apr 17 16:31:45.318806 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.318754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.318961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.318812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.318961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.318864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj7g\" (UniqueName: \"kubernetes.io/projected/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-kube-api-access-vsj7g\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.318961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.318926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-tmp-dir\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.319089 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.318988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-config-volume\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.319089 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.319012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbh2c\" (UniqueName: \"kubernetes.io/projected/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-kube-api-access-sbh2c\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.385289 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.385249    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:45.385467 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.385315    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:45.385467 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.385346    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:45.388049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388025    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bq7nv\""
Apr 17 16:31:45.388185 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388166    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 17 16:31:45.388287 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388270    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 17 16:31:45.388399 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388382    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z6rg9\""
Apr 17 16:31:45.388477 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388423    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 17 16:31:45.388477 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.388443    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 17 16:31:45.420176 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420154    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.420276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420196    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.420276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420224    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj7g\" (UniqueName: \"kubernetes.io/projected/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-kube-api-access-vsj7g\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.420276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420249    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-tmp-dir\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.420406 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.420291    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:31:45.420406 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420330    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-config-volume\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.420406 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.420357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:31:45.920342684 +0000 UTC m=+33.161001103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:31:45.420406 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420395    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbh2c\" (UniqueName: \"kubernetes.io/projected/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-kube-api-access-sbh2c\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.420582 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.420562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-tmp-dir\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.420666 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.420650    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:31:45.420738 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.420726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:31:45.920709978 +0000 UTC m=+33.161368405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:31:45.428677 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.428644    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-config-volume\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.433058 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.433038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj7g\" (UniqueName: \"kubernetes.io/projected/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-kube-api-access-vsj7g\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.440908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.440886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbh2c\" (UniqueName: \"kubernetes.io/projected/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-kube-api-access-sbh2c\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.925621 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.925589    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:45.925824 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:45.925639    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:45.925824 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.925749    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:31:45.925824 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.925787    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:31:45.925941 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.925837    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:31:46.925817029 +0000 UTC m=+34.166475448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:31:45.925941 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:45.925856    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:31:46.925847402 +0000 UTC m=+34.166505822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:31:46.026920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.026886    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:31:46.027086 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.027035    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 17 16:31:46.027157 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.027107    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:32:18.027090662 +0000 UTC m=+65.267749096 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : secret "metrics-daemon-secret" not found
Apr 17 16:31:46.229103 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.229032    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:46.231463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.231445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r26\" (UniqueName: \"kubernetes.io/projected/e9dc640a-deef-48d0-8af4-5279f7023a59-kube-api-access-d8r26\") pod \"network-check-target-t84cs\" (UID: \"e9dc640a-deef-48d0-8af4-5279f7023a59\") " pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:46.303525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.303496    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:46.449379 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.449201    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t84cs"]
Apr 17 16:31:46.483228 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:46.483153    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dc640a_deef_48d0_8af4_5279f7023a59.slice/crio-536e753dde00c52fc8098399159dab4b792b5412d043b52f8915690ed7d51780 WatchSource:0}: Error finding container 536e753dde00c52fc8098399159dab4b792b5412d043b52f8915690ed7d51780: Status 404 returned error can't find the container with id 536e753dde00c52fc8098399159dab4b792b5412d043b52f8915690ed7d51780
Apr 17 16:31:46.626994 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.626949    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t84cs" event={"ID":"e9dc640a-deef-48d0-8af4-5279f7023a59","Type":"ContainerStarted","Data":"536e753dde00c52fc8098399159dab4b792b5412d043b52f8915690ed7d51780"}
Apr 17 16:31:46.933436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.933401    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:46.933436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:46.933441    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:46.933637 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.933554    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:31:46.933637 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.933625    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:31:48.933609806 +0000 UTC m=+36.174268224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:31:46.933714 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.933561    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:31:46.933714 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:46.933669    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:31:48.933658806 +0000 UTC m=+36.174317220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:31:47.631444 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:47.631397    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="ccde3374c420b1c0d0e6152cde97db8985a8dc22d4fd67ec7cd5c8df98152bc5" exitCode=0
Apr 17 16:31:47.631885 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:47.631482    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"ccde3374c420b1c0d0e6152cde97db8985a8dc22d4fd67ec7cd5c8df98152bc5"}
Apr 17 16:31:48.636882 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:48.636844    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab" containerID="6e70e4e4203380527d8ce44fdfbf21aec50aa082aca5774c26050074858f1d4f" exitCode=0
Apr 17 16:31:48.637367 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:48.636920    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerDied","Data":"6e70e4e4203380527d8ce44fdfbf21aec50aa082aca5774c26050074858f1d4f"}
Apr 17 16:31:48.949608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:48.949526    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:48.949608 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:48.949587    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:48.949855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:48.949688    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:31:48.949855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:48.949702    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:31:48.949855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:48.949782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:31:52.949745916 +0000 UTC m=+40.190404345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:31:48.949855 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:48.949804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:31:52.94979419 +0000 UTC m=+40.190452611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:31:49.641977 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:49.641738    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25sc2" event={"ID":"0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab","Type":"ContainerStarted","Data":"af7c0f3b652e814c4dee2864065c700ad262405dbb63907b6ac412d69c9e1288"}
Apr 17 16:31:49.643074 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:49.643051    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t84cs" event={"ID":"e9dc640a-deef-48d0-8af4-5279f7023a59","Type":"ContainerStarted","Data":"3102161b14d8274696eb5148c87a2a2a6e61a4f447b00e34f4f0683bb0e2f599"}
Apr 17 16:31:49.643191 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:49.643166    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:31:49.662821 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:49.662780    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-25sc2" podStartSLOduration=4.74503982 podStartE2EDuration="36.662751159s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:14.603748588 +0000 UTC m=+1.844407005" lastFinishedPulling="2026-04-17 16:31:46.521459919 +0000 UTC m=+33.762118344" observedRunningTime="2026-04-17 16:31:49.661959227 +0000 UTC m=+36.902617663" watchObservedRunningTime="2026-04-17 16:31:49.662751159 +0000 UTC m=+36.903409588"
Apr 17 16:31:49.677639 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:49.677598    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t84cs" podStartSLOduration=33.67455441 podStartE2EDuration="36.677588517s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:31:46.500253959 +0000 UTC m=+33.740912372" lastFinishedPulling="2026-04-17 16:31:49.503288064 +0000 UTC m=+36.743946479" observedRunningTime="2026-04-17 16:31:49.677267688 +0000 UTC m=+36.917926128" watchObservedRunningTime="2026-04-17 16:31:49.677588517 +0000 UTC m=+36.918247004"
Apr 17 16:31:52.979249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:52.979211    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:31:52.979249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:52.979256    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:31:52.979756 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:52.979363    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:31:52.979756 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:52.979370    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:31:52.979756 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:52.979416    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:32:00.979403217 +0000 UTC m=+48.220061631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:31:52.979756 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:31:52.979435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:32:00.979421745 +0000 UTC m=+48.220080160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:31:55.192808 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:55.192744    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:55.196886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:55.196860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95e5b703-00aa-4d7f-b79a-e7bd129e8ce7-original-pull-secret\") pod \"global-pull-secret-syncer-fgtpp\" (UID: \"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7\") " pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:55.296119 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:55.296092    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgtpp"
Apr 17 16:31:55.428929 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:55.428897    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fgtpp"]
Apr 17 16:31:55.432298 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:55.432264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e5b703_00aa_4d7f_b79a_e7bd129e8ce7.slice/crio-1f6e85f9762e7479040650842e2f610923e2db87f71ce4c71aa8285cc70b3694 WatchSource:0}: Error finding container 1f6e85f9762e7479040650842e2f610923e2db87f71ce4c71aa8285cc70b3694: Status 404 returned error can't find the container with id 1f6e85f9762e7479040650842e2f610923e2db87f71ce4c71aa8285cc70b3694
Apr 17 16:31:55.655773 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:55.655729    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fgtpp" event={"ID":"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7","Type":"ContainerStarted","Data":"1f6e85f9762e7479040650842e2f610923e2db87f71ce4c71aa8285cc70b3694"}
Apr 17 16:31:56.909525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.909491    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"]
Apr 17 16:31:56.937508 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.937478    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"]
Apr 17 16:31:56.937641 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.937581    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:56.939873 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.939847    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j58ph\""
Apr 17 16:31:56.940011 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.939973    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Apr 17 16:31:56.940630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.940613    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Apr 17 16:31:56.940630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.940624    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Apr 17 16:31:56.940779 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:56.940634    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Apr 17 16:31:57.106575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.106540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d30abe04-de55-4600-af4f-e8aaf3512470-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.106575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.106578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwkp\" (UniqueName: \"kubernetes.io/projected/d30abe04-de55-4600-af4f-e8aaf3512470-kube-api-access-lfwkp\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.207836 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.207737    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d30abe04-de55-4600-af4f-e8aaf3512470-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.207836 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.207788    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwkp\" (UniqueName: \"kubernetes.io/projected/d30abe04-de55-4600-af4f-e8aaf3512470-kube-api-access-lfwkp\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.210267 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.210242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d30abe04-de55-4600-af4f-e8aaf3512470-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.215198 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.215180    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwkp\" (UniqueName: \"kubernetes.io/projected/d30abe04-de55-4600-af4f-e8aaf3512470-kube-api-access-lfwkp\") pod \"managed-serviceaccount-addon-agent-667468957f-qkl69\" (UID: \"d30abe04-de55-4600-af4f-e8aaf3512470\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.258304 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.258284    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"
Apr 17 16:31:57.364198 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.364171    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69"]
Apr 17 16:31:57.367373 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:31:57.367349    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd30abe04_de55_4600_af4f_e8aaf3512470.slice/crio-962c7245cc73d483f130eef6aeef56eff806cef6bbb0fce3397069ce97ff188f WatchSource:0}: Error finding container 962c7245cc73d483f130eef6aeef56eff806cef6bbb0fce3397069ce97ff188f: Status 404 returned error can't find the container with id 962c7245cc73d483f130eef6aeef56eff806cef6bbb0fce3397069ce97ff188f
Apr 17 16:31:57.660405 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:31:57.660370    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69" event={"ID":"d30abe04-de55-4600-af4f-e8aaf3512470","Type":"ContainerStarted","Data":"962c7245cc73d483f130eef6aeef56eff806cef6bbb0fce3397069ce97ff188f"}
Apr 17 16:32:01.037986 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:01.037903    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:32:01.037986 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:01.037945    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:32:01.038409 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:01.038049    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:32:01.038409 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:01.038114    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:32:17.038095545 +0000 UTC m=+64.278753964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:32:01.038409 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:01.038055    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:32:01.038409 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:01.038177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:32:17.038164916 +0000 UTC m=+64.278823329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:32:01.668925 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:01.668882    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69" event={"ID":"d30abe04-de55-4600-af4f-e8aaf3512470","Type":"ContainerStarted","Data":"1c706595909a4882b5f377035d4bd479fdd27300818f493d35d29c78b2c9b994"}
Apr 17 16:32:01.683703 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:01.683652    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69" podStartSLOduration=2.41807041 podStartE2EDuration="5.68363965s" podCreationTimestamp="2026-04-17 16:31:56 +0000 UTC" firstStartedPulling="2026-04-17 16:31:57.36925692 +0000 UTC m=+44.609915334" lastFinishedPulling="2026-04-17 16:32:00.634826156 +0000 UTC m=+47.875484574" observedRunningTime="2026-04-17 16:32:01.683305541 +0000 UTC m=+48.923963977" watchObservedRunningTime="2026-04-17 16:32:01.68363965 +0000 UTC m=+48.924298087"
Apr 17 16:32:05.603229 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:05.603187    2579 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0"
Apr 17 16:32:05.603649 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:05.603361    2579 kuberuntime_manager.go:1358] "Unhandled Error" err="container &Container{Name:global-pull-secret-syncer,Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0,Command:[/usr/bin/control-plane-operator],Args:[sync-global-pullsecret],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{5 -3} {<nil>} 5m DecimalSI},memory: {{36700160 0} {<nil>} 35Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kubelet-config,ReadOnly:false,MountPath:/var/lib/kubelet,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:dbus,ReadOnly:false,MountPath:/var/run/dbus,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:original-pull-secret,ReadOnly:true,MountPath:/etc/original-pull-secret,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod global-pull-secret-syncer-fgtpp_kube-system(95e5b703-00aa-4d7f-b79a-e7bd129e8ce7): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Apr 17 16:32:05.604538 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:05.604508    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"global-pull-secret-syncer\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:32:05.676754 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:05.676724    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"global-pull-secret-syncer\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86fd02d09596be124562146df9ab5bf33cd7cdfde29f701524b250a0e8beec0: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="kube-system/global-pull-secret-syncer-fgtpp" podUID="95e5b703-00aa-4d7f-b79a-e7bd129e8ce7"
Apr 17 16:32:10.628945 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:10.628913    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4rx"
Apr 17 16:32:17.128719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:17.128676    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:32:17.128719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:17.128722    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:32:17.129226 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:17.128835    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:32:17.129226 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:17.128855    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:32:17.129226 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:17.128902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:32:49.12888918 +0000 UTC m=+96.369547594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:32:17.129226 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:17.128915    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:32:49.128909246 +0000 UTC m=+96.369567660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:32:18.034484 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:18.034448    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:32:18.034652 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:18.034617    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 17 16:32:18.034704 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:18.034688    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:22.034669046 +0000 UTC m=+129.275327474 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : secret "metrics-daemon-secret" not found
Apr 17 16:32:20.648632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:20.648598    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t84cs"
Apr 17 16:32:22.709208 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:22.709168    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fgtpp" event={"ID":"95e5b703-00aa-4d7f-b79a-e7bd129e8ce7","Type":"ContainerStarted","Data":"b39b0d8882a8cdf18c7c6ba08ddaa7ada6918ccea2567fa67f8d24fb5b0edb90"}
Apr 17 16:32:22.725924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:22.725872    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fgtpp" podStartSLOduration=33.205936359 podStartE2EDuration="59.725859106s" podCreationTimestamp="2026-04-17 16:31:23 +0000 UTC" firstStartedPulling="2026-04-17 16:31:55.434311946 +0000 UTC m=+42.674970361" lastFinishedPulling="2026-04-17 16:32:21.954234689 +0000 UTC m=+69.194893108" observedRunningTime="2026-04-17 16:32:22.725441054 +0000 UTC m=+69.966099490" watchObservedRunningTime="2026-04-17 16:32:22.725859106 +0000 UTC m=+69.966517542"
Apr 17 16:32:49.147569 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:49.147515    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:32:49.148025 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:32:49.147614    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:32:49.148025 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:49.147679    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:32:49.148025 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:49.147692    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 17 16:32:49.148025 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:49.147748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls podName:2e7b194d-150a-4fd7-9f3d-e8475c1cd65d nodeName:}" failed. No retries permitted until 2026-04-17 16:33:53.147730767 +0000 UTC m=+160.388389187 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls") pod "dns-default-f8sph" (UID: "2e7b194d-150a-4fd7-9f3d-e8475c1cd65d") : secret "dns-default-metrics-tls" not found
Apr 17 16:32:49.148025 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:32:49.147785    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:33:53.147757092 +0000 UTC m=+160.388415506 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:33:22.076838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:22.076788    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:33:22.077315 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:22.076931    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 17 16:33:22.077315 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:22.077010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs podName:e4163dd4-e68b-4ae1-818d-81e2a6e33a15 nodeName:}" failed. No retries permitted until 2026-04-17 16:35:24.076995342 +0000 UTC m=+251.317653761 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs") pod "network-metrics-daemon-fwn9m" (UID: "e4163dd4-e68b-4ae1-818d-81e2a6e33a15") : secret "metrics-daemon-secret" not found
Apr 17 16:33:39.267727 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.267693    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"]
Apr 17 16:33:39.270530 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.270508    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"
Apr 17 16:33:39.270958 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.270938    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-585dfdc468-k28wq"]
Apr 17 16:33:39.272903 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.272885    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.272995 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.272885    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.272995 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.272965    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tmv7x\""
Apr 17 16:33:39.273412 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.273399    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"]
Apr 17 16:33:39.273540 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.273526    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.275690 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.275674    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4zbmj\""
Apr 17 16:33:39.275838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.275720    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.275838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.275804    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.276013 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.275994    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-vnhv5"]
Apr 17 16:33:39.276092 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.276063    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Apr 17 16:33:39.276092 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.276083    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Apr 17 16:33:39.276331 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.276313    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.278680 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.278658    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.279357 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.279338    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.280109 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.280084    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Apr 17 16:33:39.283649 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.283625    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Apr 17 16:33:39.283944 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.283925    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.284169 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284145    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8htfp\""
Apr 17 16:33:39.284169 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284152    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.284317 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284170    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wxg92\""
Apr 17 16:33:39.284430 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284416    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Apr 17 16:33:39.284542 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284525    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.284687 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.284653    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Apr 17 16:33:39.285812 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.285376    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:33:39.288205 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.288183    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Apr 17 16:33:39.288349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.288295    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.288414 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.288197    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"]
Apr 17 16:33:39.290658 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.290640    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nd5dg\""
Apr 17 16:33:39.290755 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.290709    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Apr 17 16:33:39.290975 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.290957    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Apr 17 16:33:39.291142 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.291122    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Apr 17 16:33:39.291215 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.291164    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Apr 17 16:33:39.296606 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.296586    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-585dfdc468-k28wq"]
Apr 17 16:33:39.297609 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.297589    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-vnhv5"]
Apr 17 16:33:39.298240 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.298220    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Apr 17 16:33:39.298551 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.298534    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"]
Apr 17 16:33:39.302057 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.302038    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:33:39.371415 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.371379    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"]
Apr 17 16:33:39.374354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.374324    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.375173 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.375153    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-574d8f6f84-q4xkf"]
Apr 17 16:33:39.376508 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.376491    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Apr 17 16:33:39.376880 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.376861    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.377233 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.377216    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5czhm\""
Apr 17 16:33:39.377454 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.377437    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Apr 17 16:33:39.377574 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.377560    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.377949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.377935    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.380734 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.380713    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Apr 17 16:33:39.380847 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.380799    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Apr 17 16:33:39.380847 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.380838    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Apr 17 16:33:39.381225 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.381205    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Apr 17 16:33:39.381325 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.381258    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Apr 17 16:33:39.381386 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.381331    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Apr 17 16:33:39.381386 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.381265    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9w4pd\""
Apr 17 16:33:39.391523 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.391437    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"]
Apr 17 16:33:39.391630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.391566    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-574d8f6f84-q4xkf"]
Apr 17 16:33:39.402897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-snapshots\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.402897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-serving-cert\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.402897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57050c6f-fb02-47e6-96b0-58ebe611a2bb-serving-cert\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.402897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.402897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dfe89383-3a58-4c17-847d-fe8456a068e8-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgpw\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.402978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf7d\" (UniqueName: \"kubernetes.io/projected/57050c6f-fb02-47e6-96b0-58ebe611a2bb-kube-api-access-2qf7d\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-trusted-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpcj\" (UniqueName: \"kubernetes.io/projected/dfe89383-3a58-4c17-847d-fe8456a068e8-kube-api-access-rnpcj\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.403217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfxn\" (UniqueName: \"kubernetes.io/projected/98f6bd2e-c563-4fc2-b391-a03bd570bcbc-kube-api-access-sqfxn\") pod \"volume-data-source-validator-7c6cbb6c87-sdbzt\" (UID: \"98f6bd2e-c563-4fc2-b391-a03bd570bcbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-service-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-config\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-trusted-ca\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-tmp\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlpz\" (UniqueName: \"kubernetes.io/projected/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-kube-api-access-bjlpz\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.403526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.403816 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.403581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.504821 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504784    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwn5z\" (UniqueName: \"kubernetes.io/projected/9884d527-3a7b-4c87-87d8-3df862c613e2-kube-api-access-pwn5z\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504868    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-trusted-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504888    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpcj\" (UniqueName: \"kubernetes.io/projected/dfe89383-3a58-4c17-847d-fe8456a068e8-kube-api-access-rnpcj\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504911    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfxn\" (UniqueName: \"kubernetes.io/projected/98f6bd2e-c563-4fc2-b391-a03bd570bcbc-kube-api-access-sqfxn\") pod \"volume-data-source-validator-7c6cbb6c87-sdbzt\" (UID: \"98f6bd2e-c563-4fc2-b391-a03bd570bcbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.505028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.504996    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505076    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-service-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505109    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505135    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-default-certificate\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505194    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-config\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.505218    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.505241    2579 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.505253    2579 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f578588c7-6jgxw: secret "image-registry-tls" not found
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.505340    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls podName:bc76c219-e0ad-4b46-ab84-19137ac2d81e nodeName:}" failed. No retries permitted until 2026-04-17 16:33:40.005320584 +0000 UTC m=+147.245979002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls") pod "image-registry-6f578588c7-6jgxw" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e") : secret "image-registry-tls" not found
Apr 17 16:33:39.505377 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.505356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:40.005347171 +0000 UTC m=+147.246005588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9884d527-3a7b-4c87-87d8-3df862c613e2-config\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505421    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-trusted-ca\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505475    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-tmp\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505505    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlpz\" (UniqueName: \"kubernetes.io/projected/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-kube-api-access-bjlpz\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505535    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505568    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxbc\" (UniqueName: \"kubernetes.io/projected/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-kube-api-access-wqxbc\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505632    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-snapshots\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505658    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-serving-cert\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505681    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57050c6f-fb02-47e6-96b0-58ebe611a2bb-serving-cert\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505714    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505742    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-service-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-tmp\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-stats-auth\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.505922 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505859    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dfe89383-3a58-4c17-847d-fe8456a068e8-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505906    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgpw\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505922    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-config\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505933    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-trusted-ca-bundle\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.505938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9884d527-3a7b-4c87-87d8-3df862c613e2-serving-cert\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506002    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506041    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf7d\" (UniqueName: \"kubernetes.io/projected/57050c6f-fb02-47e6-96b0-58ebe611a2bb-kube-api-access-2qf7d\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506281    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57050c6f-fb02-47e6-96b0-58ebe611a2bb-trusted-ca\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-snapshots\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.506688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506614    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dfe89383-3a58-4c17-847d-fe8456a068e8-telemetry-config\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.507237 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.506724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.508533 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.508506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-serving-cert\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.508652 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.508591    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.508812 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.508794    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57050c6f-fb02-47e6-96b0-58ebe611a2bb-serving-cert\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.508962 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.508945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.525671 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.525610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlpz\" (UniqueName: \"kubernetes.io/projected/047abce8-32b2-4b6d-8f0e-2c7ce6ef6401-kube-api-access-bjlpz\") pod \"insights-operator-585dfdc468-k28wq\" (UID: \"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401\") " pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.526157 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.526130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf7d\" (UniqueName: \"kubernetes.io/projected/57050c6f-fb02-47e6-96b0-58ebe611a2bb-kube-api-access-2qf7d\") pod \"console-operator-9d4b6777b-vnhv5\" (UID: \"57050c6f-fb02-47e6-96b0-58ebe611a2bb\") " pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.526275 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.526193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.526275 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.526192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgpw\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:39.526709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.526689    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfxn\" (UniqueName: \"kubernetes.io/projected/98f6bd2e-c563-4fc2-b391-a03bd570bcbc-kube-api-access-sqfxn\") pod \"volume-data-source-validator-7c6cbb6c87-sdbzt\" (UID: \"98f6bd2e-c563-4fc2-b391-a03bd570bcbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"
Apr 17 16:33:39.527451 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.527432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpcj\" (UniqueName: \"kubernetes.io/projected/dfe89383-3a58-4c17-847d-fe8456a068e8-kube-api-access-rnpcj\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:39.584344 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.584296    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"
Apr 17 16:33:39.591819 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.591789    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-585dfdc468-k28wq"
Apr 17 16:33:39.605724 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.605697    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:39.606488 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606355    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-stats-auth\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.606488 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606397    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9884d527-3a7b-4c87-87d8-3df862c613e2-serving-cert\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.606488 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606479    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pwn5z\" (UniqueName: \"kubernetes.io/projected/9884d527-3a7b-4c87-87d8-3df862c613e2-kube-api-access-pwn5z\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.606665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606528    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.606665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606599    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-default-certificate\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.606665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606633    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9884d527-3a7b-4c87-87d8-3df862c613e2-config\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.606826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606690    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.606826 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.606728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:40.106704713 +0000 UTC m=+147.347363143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : configmap references non-existent config key: service-ca.crt
Apr 17 16:33:39.606826 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.606799    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Apr 17 16:33:39.606826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.606805    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxbc\" (UniqueName: \"kubernetes.io/projected/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-kube-api-access-wqxbc\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.607074 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:39.606863    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:40.106844776 +0000 UTC m=+147.347503196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : secret "router-metrics-certs-default" not found
Apr 17 16:33:39.608225 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.608205    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9884d527-3a7b-4c87-87d8-3df862c613e2-config\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.609243 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.609217    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-default-certificate\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.609869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.609833    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-stats-auth\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.611148 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.611110    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9884d527-3a7b-4c87-87d8-3df862c613e2-serving-cert\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.616447 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.615840    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxbc\" (UniqueName: \"kubernetes.io/projected/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-kube-api-access-wqxbc\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:39.616447 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.615880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwn5z\" (UniqueName: \"kubernetes.io/projected/9884d527-3a7b-4c87-87d8-3df862c613e2-kube-api-access-pwn5z\") pod \"service-ca-operator-d6fc45fc5-2lq2h\" (UID: \"9884d527-3a7b-4c87-87d8-3df862c613e2\") " pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.684458 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.684141    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"
Apr 17 16:33:39.728416 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.728381    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt"]
Apr 17 16:33:39.747967 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.747934    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-585dfdc468-k28wq"]
Apr 17 16:33:39.751720 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:39.751695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047abce8_32b2_4b6d_8f0e_2c7ce6ef6401.slice/crio-7523a2fd6adf7a29096bc4de45cf07661bd91eee6d92d39d20497e1c081c6e29 WatchSource:0}: Error finding container 7523a2fd6adf7a29096bc4de45cf07661bd91eee6d92d39d20497e1c081c6e29: Status 404 returned error can't find the container with id 7523a2fd6adf7a29096bc4de45cf07661bd91eee6d92d39d20497e1c081c6e29
Apr 17 16:33:39.764469 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.764441    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-vnhv5"]
Apr 17 16:33:39.768212 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:39.768187    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57050c6f_fb02_47e6_96b0_58ebe611a2bb.slice/crio-0b4e8660137b26555499a80576bb290de3fdf01ed5944909de07a4a1a46ecc53 WatchSource:0}: Error finding container 0b4e8660137b26555499a80576bb290de3fdf01ed5944909de07a4a1a46ecc53: Status 404 returned error can't find the container with id 0b4e8660137b26555499a80576bb290de3fdf01ed5944909de07a4a1a46ecc53
Apr 17 16:33:39.808068 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.807997    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h"]
Apr 17 16:33:39.810533 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:39.810510    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9884d527_3a7b_4c87_87d8_3df862c613e2.slice/crio-0e2d71d008f2f06ebbeed5f8a1a2e24c7099ef35865cde06acfcc219a83f28fc WatchSource:0}: Error finding container 0e2d71d008f2f06ebbeed5f8a1a2e24c7099ef35865cde06acfcc219a83f28fc: Status 404 returned error can't find the container with id 0e2d71d008f2f06ebbeed5f8a1a2e24c7099ef35865cde06acfcc219a83f28fc
Apr 17 16:33:39.859100 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.859066    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-585dfdc468-k28wq" event={"ID":"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401","Type":"ContainerStarted","Data":"7523a2fd6adf7a29096bc4de45cf07661bd91eee6d92d39d20497e1c081c6e29"}
Apr 17 16:33:39.859990 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.859967    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" event={"ID":"57050c6f-fb02-47e6-96b0-58ebe611a2bb","Type":"ContainerStarted","Data":"0b4e8660137b26555499a80576bb290de3fdf01ed5944909de07a4a1a46ecc53"}
Apr 17 16:33:39.860916 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.860898    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h" event={"ID":"9884d527-3a7b-4c87-87d8-3df862c613e2","Type":"ContainerStarted","Data":"0e2d71d008f2f06ebbeed5f8a1a2e24c7099ef35865cde06acfcc219a83f28fc"}
Apr 17 16:33:39.861753 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:39.861728    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt" event={"ID":"98f6bd2e-c563-4fc2-b391-a03bd570bcbc","Type":"ContainerStarted","Data":"7c46cc7e2df57163b42b87f3f27c0a55dc79c581acda8aa43112d2dcafa003b6"}
Apr 17 16:33:40.010076 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:40.010040    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:40.010076 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:40.010073    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:40.010282 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.010184    2579 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 17 16:33:40.010282 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.010185    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:40.010282 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.010252    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:41.010236796 +0000 UTC m=+148.250895216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:40.010282 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.010195    2579 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f578588c7-6jgxw: secret "image-registry-tls" not found
Apr 17 16:33:40.010412 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.010314    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls podName:bc76c219-e0ad-4b46-ab84-19137ac2d81e nodeName:}" failed. No retries permitted until 2026-04-17 16:33:41.010303823 +0000 UTC m=+148.250962238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls") pod "image-registry-6f578588c7-6jgxw" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e") : secret "image-registry-tls" not found
Apr 17 16:33:40.111227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:40.111141    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:40.111227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:40.111206    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:40.111411 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.111302    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Apr 17 16:33:40.111411 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.111305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:41.11129059 +0000 UTC m=+148.351949025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : configmap references non-existent config key: service-ca.crt
Apr 17 16:33:40.111411 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:40.111362    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:41.111349328 +0000 UTC m=+148.352007745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : secret "router-metrics-certs-default" not found
Apr 17 16:33:41.020657 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:41.020516    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:41.020657 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:41.020575    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:41.020657 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.020644    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:41.021232 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.020708    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:43.020690992 +0000 UTC m=+150.261349415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:41.021232 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.020731    2579 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 17 16:33:41.021232 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.020771    2579 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f578588c7-6jgxw: secret "image-registry-tls" not found
Apr 17 16:33:41.021232 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.020826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls podName:bc76c219-e0ad-4b46-ab84-19137ac2d81e nodeName:}" failed. No retries permitted until 2026-04-17 16:33:43.020808246 +0000 UTC m=+150.261466678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls") pod "image-registry-6f578588c7-6jgxw" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e") : secret "image-registry-tls" not found
Apr 17 16:33:41.122588 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:41.121956    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:41.122588 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:41.122049    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:41.122588 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.122185    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Apr 17 16:33:41.122588 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.122253    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:43.122233973 +0000 UTC m=+150.362892393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : secret "router-metrics-certs-default" not found
Apr 17 16:33:41.122588 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:41.122297    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:43.122286545 +0000 UTC m=+150.362944963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : configmap references non-existent config key: service-ca.crt
Apr 17 16:33:42.870301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.870245    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h" event={"ID":"9884d527-3a7b-4c87-87d8-3df862c613e2","Type":"ContainerStarted","Data":"00ac58129096d6a88111544652512a8723122b3dfe0c25141e122e39cd0d933e"}
Apr 17 16:33:42.871789 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.871729    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt" event={"ID":"98f6bd2e-c563-4fc2-b391-a03bd570bcbc","Type":"ContainerStarted","Data":"09f59f4daed7cc7391bf3e039087a51ccdaf84b4f8e2519d64f9cbf6c793dc0c"}
Apr 17 16:33:42.873195 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.873157    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-585dfdc468-k28wq" event={"ID":"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401","Type":"ContainerStarted","Data":"a1233d8fb9ce66690ad18291a0e82161271f7062dad2193bf4b229b69ec033d9"}
Apr 17 16:33:42.875516 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.875491    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" event={"ID":"57050c6f-fb02-47e6-96b0-58ebe611a2bb","Type":"ContainerStarted","Data":"31b087b1240fc7789f3bc2bf8df06d82833e22dbdd577b00d948e882e3f2de50"}
Apr 17 16:33:42.875682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.875667    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:42.877226 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.876966    2579 patch_prober.go:28] interesting pod/console-operator-9d4b6777b-vnhv5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Apr 17 16:33:42.877226 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.877013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podUID="57050c6f-fb02-47e6-96b0-58ebe611a2bb" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused"
Apr 17 16:33:42.892622 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.891879    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h" podStartSLOduration=0.961020433 podStartE2EDuration="3.89186623s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="2026-04-17 16:33:39.812214169 +0000 UTC m=+147.052872584" lastFinishedPulling="2026-04-17 16:33:42.743059953 +0000 UTC m=+149.983718381" observedRunningTime="2026-04-17 16:33:42.89108035 +0000 UTC m=+150.131738785" watchObservedRunningTime="2026-04-17 16:33:42.89186623 +0000 UTC m=+150.132524665"
Apr 17 16:33:42.906053 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.906004    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-7c6cbb6c87-sdbzt" podStartSLOduration=0.908875632 podStartE2EDuration="3.905986128s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="2026-04-17 16:33:39.734558263 +0000 UTC m=+146.975216680" lastFinishedPulling="2026-04-17 16:33:42.731668747 +0000 UTC m=+149.972327176" observedRunningTime="2026-04-17 16:33:42.905499723 +0000 UTC m=+150.146158153" watchObservedRunningTime="2026-04-17 16:33:42.905986128 +0000 UTC m=+150.146644565"
Apr 17 16:33:42.920266 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.920223    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-585dfdc468-k28wq" podStartSLOduration=0.93832436 podStartE2EDuration="3.92021102s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="2026-04-17 16:33:39.753813295 +0000 UTC m=+146.994471708" lastFinishedPulling="2026-04-17 16:33:42.735699941 +0000 UTC m=+149.976358368" observedRunningTime="2026-04-17 16:33:42.919470354 +0000 UTC m=+150.160128784" watchObservedRunningTime="2026-04-17 16:33:42.92021102 +0000 UTC m=+150.160869455"
Apr 17 16:33:42.936000 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:42.935949    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podStartSLOduration=0.967435845 podStartE2EDuration="3.93593259s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="2026-04-17 16:33:39.770151662 +0000 UTC m=+147.010810081" lastFinishedPulling="2026-04-17 16:33:42.738648409 +0000 UTC m=+149.979306826" observedRunningTime="2026-04-17 16:33:42.934573324 +0000 UTC m=+150.175231759" watchObservedRunningTime="2026-04-17 16:33:42.93593259 +0000 UTC m=+150.176591027"
Apr 17 16:33:43.038487 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.038389    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:43.038487 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.038442    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:43.038711 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.038529    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:43.038711 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.038599    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:47.038584804 +0000 UTC m=+154.279243218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:43.038711 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.038614    2579 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 17 16:33:43.038711 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.038633    2579 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f578588c7-6jgxw: secret "image-registry-tls" not found
Apr 17 16:33:43.038711 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.038679    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls podName:bc76c219-e0ad-4b46-ab84-19137ac2d81e nodeName:}" failed. No retries permitted until 2026-04-17 16:33:47.038663728 +0000 UTC m=+154.279322147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls") pod "image-registry-6f578588c7-6jgxw" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e") : secret "image-registry-tls" not found
Apr 17 16:33:43.139565 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.139525    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:43.139737 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.139587    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:43.139737 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.139693    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Apr 17 16:33:43.139737 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.139719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:47.1396964 +0000 UTC m=+154.380354828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : configmap references non-existent config key: service-ca.crt
Apr 17 16:33:43.139878 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:43.139746    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:47.139736535 +0000 UTC m=+154.380394952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : secret "router-metrics-certs-default" not found
Apr 17 16:33:43.879083 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.879054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/0.log"
Apr 17 16:33:43.879581 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.879093    2579 generic.go:358] "Generic (PLEG): container finished" podID="57050c6f-fb02-47e6-96b0-58ebe611a2bb" containerID="31b087b1240fc7789f3bc2bf8df06d82833e22dbdd577b00d948e882e3f2de50" exitCode=255
Apr 17 16:33:43.879581 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.879125    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" event={"ID":"57050c6f-fb02-47e6-96b0-58ebe611a2bb","Type":"ContainerDied","Data":"31b087b1240fc7789f3bc2bf8df06d82833e22dbdd577b00d948e882e3f2de50"}
Apr 17 16:33:43.879581 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:43.879473    2579 scope.go:117] "RemoveContainer" containerID="31b087b1240fc7789f3bc2bf8df06d82833e22dbdd577b00d948e882e3f2de50"
Apr 17 16:33:44.882770 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.882741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:33:44.883168 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.883153    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/0.log"
Apr 17 16:33:44.883215 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.883187    2579 generic.go:358] "Generic (PLEG): container finished" podID="57050c6f-fb02-47e6-96b0-58ebe611a2bb" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6" exitCode=255
Apr 17 16:33:44.883246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.883235    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" event={"ID":"57050c6f-fb02-47e6-96b0-58ebe611a2bb","Type":"ContainerDied","Data":"8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"}
Apr 17 16:33:44.883282 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.883261    2579 scope.go:117] "RemoveContainer" containerID="31b087b1240fc7789f3bc2bf8df06d82833e22dbdd577b00d948e882e3f2de50"
Apr 17 16:33:44.883531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:44.883504    2579 scope.go:117] "RemoveContainer" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"
Apr 17 16:33:44.883706 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:44.883685    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-vnhv5_openshift-console-operator(57050c6f-fb02-47e6-96b0-58ebe611a2bb)\"" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podUID="57050c6f-fb02-47e6-96b0-58ebe611a2bb"
Apr 17 16:33:45.887094 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:45.887064    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:33:45.887498 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:45.887409    2579 scope.go:117] "RemoveContainer" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"
Apr 17 16:33:45.887599 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:45.887576    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-vnhv5_openshift-console-operator(57050c6f-fb02-47e6-96b0-58ebe611a2bb)\"" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podUID="57050c6f-fb02-47e6-96b0-58ebe611a2bb"
Apr 17 16:33:46.793687 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.793661    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j5cpk_38c0cc14-3fcb-4193-8823-9a4e0a06f777/dns-node-resolver/0.log"
Apr 17 16:33:46.848605 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.848575    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-865cb79987-rsc72"]
Apr 17 16:33:46.852830 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.852813    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.855022 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.855002    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Apr 17 16:33:46.855022 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.855011    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Apr 17 16:33:46.855195 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.855012    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k9vf2\""
Apr 17 16:33:46.855195 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.855060    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Apr 17 16:33:46.855195 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.855158    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Apr 17 16:33:46.858230 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.858208    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-rsc72"]
Apr 17 16:33:46.870972 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.870946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7530e863-c13a-47d5-a670-640bb258549d-signing-cabundle\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.871088 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.870989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7530e863-c13a-47d5-a670-640bb258549d-signing-key\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.871088 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.871041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlk2\" (UniqueName: \"kubernetes.io/projected/7530e863-c13a-47d5-a670-640bb258549d-kube-api-access-2zlk2\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.972029 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.971988    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlk2\" (UniqueName: \"kubernetes.io/projected/7530e863-c13a-47d5-a670-640bb258549d-kube-api-access-2zlk2\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.972486 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.972119    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7530e863-c13a-47d5-a670-640bb258549d-signing-cabundle\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.972486 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.972144    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7530e863-c13a-47d5-a670-640bb258549d-signing-key\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.972923 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.972899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7530e863-c13a-47d5-a670-640bb258549d-signing-cabundle\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.974525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.974503    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7530e863-c13a-47d5-a670-640bb258549d-signing-key\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:46.980499 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:46.980476    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlk2\" (UniqueName: \"kubernetes.io/projected/7530e863-c13a-47d5-a670-640bb258549d-kube-api-access-2zlk2\") pod \"service-ca-865cb79987-rsc72\" (UID: \"7530e863-c13a-47d5-a670-640bb258549d\") " pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:47.073502 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.073411    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:47.073502 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.073454    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:47.073712 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.073563    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:47.073712 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.073622    2579 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 17 16:33:47.073712 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.073631    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:55.073615838 +0000 UTC m=+162.314274257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:47.073712 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.073637    2579 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f578588c7-6jgxw: secret "image-registry-tls" not found
Apr 17 16:33:47.073712 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.073681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls podName:bc76c219-e0ad-4b46-ab84-19137ac2d81e nodeName:}" failed. No retries permitted until 2026-04-17 16:33:55.07366791 +0000 UTC m=+162.314326325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls") pod "image-registry-6f578588c7-6jgxw" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e") : secret "image-registry-tls" not found
Apr 17 16:33:47.162624 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.162583    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-rsc72"
Apr 17 16:33:47.174776 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.174732    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:47.174901 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.174825    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:47.174952 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.174902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:55.17488744 +0000 UTC m=+162.415545875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : configmap references non-existent config key: service-ca.crt
Apr 17 16:33:47.175000 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.174980    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Apr 17 16:33:47.175061 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:47.175049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs podName:e0783a18-1157-4c0a-83cf-e1c50ed9d2f7 nodeName:}" failed. No retries permitted until 2026-04-17 16:33:55.175029166 +0000 UTC m=+162.415687595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs") pod "router-default-574d8f6f84-q4xkf" (UID: "e0783a18-1157-4c0a-83cf-e1c50ed9d2f7") : secret "router-metrics-certs-default" not found
Apr 17 16:33:47.279504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.279471    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-rsc72"]
Apr 17 16:33:47.282919 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:47.282894    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7530e863_c13a_47d5_a670_640bb258549d.slice/crio-d67a57913d42360dab9a00134977541d748957bdd00b4cf7b4c676b6bbd12c02 WatchSource:0}: Error finding container d67a57913d42360dab9a00134977541d748957bdd00b4cf7b4c676b6bbd12c02: Status 404 returned error can't find the container with id d67a57913d42360dab9a00134977541d748957bdd00b4cf7b4c676b6bbd12c02
Apr 17 16:33:47.591440 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.591345    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76q52_4bd685f1-9eee-4a49-9473-4db1d50bd8b8/node-ca/0.log"
Apr 17 16:33:47.894268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.894238    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-rsc72" event={"ID":"7530e863-c13a-47d5-a670-640bb258549d","Type":"ContainerStarted","Data":"663cefc9de75e10f70048cfb71f6f2c10b85058c50d805b838fcee2ac69cabc9"}
Apr 17 16:33:47.894268 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.894274    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-rsc72" event={"ID":"7530e863-c13a-47d5-a670-640bb258549d","Type":"ContainerStarted","Data":"d67a57913d42360dab9a00134977541d748957bdd00b4cf7b4c676b6bbd12c02"}
Apr 17 16:33:47.917618 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:47.917570    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-865cb79987-rsc72" podStartSLOduration=1.917557087 podStartE2EDuration="1.917557087s" podCreationTimestamp="2026-04-17 16:33:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:33:47.917176352 +0000 UTC m=+155.157834788" watchObservedRunningTime="2026-04-17 16:33:47.917557087 +0000 UTC m=+155.158215581"
Apr 17 16:33:48.190367 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:48.190276    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-f8sph" podUID="2e7b194d-150a-4fd7-9f3d-e8475c1cd65d"
Apr 17 16:33:48.196277 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:48.196247    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-pmlp4" podUID="d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a"
Apr 17 16:33:48.408244 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:48.408202    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fwn9m" podUID="e4163dd4-e68b-4ae1-818d-81e2a6e33a15"
Apr 17 16:33:48.898057 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:48.898022    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f8sph"
Apr 17 16:33:49.606868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:49.606827    2579 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:49.607377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:49.607323    2579 scope.go:117] "RemoveContainer" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"
Apr 17 16:33:49.607568 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:49.607542    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-vnhv5_openshift-console-operator(57050c6f-fb02-47e6-96b0-58ebe611a2bb)\"" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podUID="57050c6f-fb02-47e6-96b0-58ebe611a2bb"
Apr 17 16:33:52.875867 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:52.875832    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:33:52.876277 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:52.876237    2579 scope.go:117] "RemoveContainer" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"
Apr 17 16:33:52.876439 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:52.876420    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-vnhv5_openshift-console-operator(57050c6f-fb02-47e6-96b0-58ebe611a2bb)\"" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" podUID="57050c6f-fb02-47e6-96b0-58ebe611a2bb"
Apr 17 16:33:53.227177 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.227087    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:33:53.227177 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.227127    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:33:53.227400 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:53.227233    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 17 16:33:53.227400 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:53.227297    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert podName:d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a nodeName:}" failed. No retries permitted until 2026-04-17 16:35:55.227284268 +0000 UTC m=+282.467942682 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert") pod "ingress-canary-pmlp4" (UID: "d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a") : secret "canary-serving-cert" not found
Apr 17 16:33:53.229516 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.229493    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e7b194d-150a-4fd7-9f3d-e8475c1cd65d-metrics-tls\") pod \"dns-default-f8sph\" (UID: \"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d\") " pod="openshift-dns/dns-default-f8sph"
Apr 17 16:33:53.401131 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.401104    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f96dz\""
Apr 17 16:33:53.410160 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.410128    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f8sph"
Apr 17 16:33:53.543427 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.543387    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f8sph"]
Apr 17 16:33:53.546533 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:53.546503    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7b194d_150a_4fd7_9f3d_e8475c1cd65d.slice/crio-e36361b218767acd3be445da656d14f7e7261376eee9975cdd5dd53d6183ca4d WatchSource:0}: Error finding container e36361b218767acd3be445da656d14f7e7261376eee9975cdd5dd53d6183ca4d: Status 404 returned error can't find the container with id e36361b218767acd3be445da656d14f7e7261376eee9975cdd5dd53d6183ca4d
Apr 17 16:33:53.911191 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:53.911153    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8sph" event={"ID":"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d","Type":"ContainerStarted","Data":"e36361b218767acd3be445da656d14f7e7261376eee9975cdd5dd53d6183ca4d"}
Apr 17 16:33:55.145730 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.145647    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:33:55.145730 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.145695    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:55.146200 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:55.145859    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:55.146200 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:33:55.145917    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls podName:dfe89383-3a58-4c17-847d-fe8456a068e8 nodeName:}" failed. No retries permitted until 2026-04-17 16:34:11.145897554 +0000 UTC m=+178.386555983 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-75587bd455-mk5cl" (UID: "dfe89383-3a58-4c17-847d-fe8456a068e8") : secret "cluster-monitoring-operator-tls" not found
Apr 17 16:33:55.148246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.148224    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"image-registry-6f578588c7-6jgxw\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") " pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:55.211597 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.211561    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:55.247090 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.247052    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:55.247263 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.247130    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:55.248749 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.247802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-service-ca-bundle\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:55.249959 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.249922    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0783a18-1157-4c0a-83cf-e1c50ed9d2f7-metrics-certs\") pod \"router-default-574d8f6f84-q4xkf\" (UID: \"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7\") " pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:55.295246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.295216    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:55.338084 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.338051    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:33:55.342809 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:55.342755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc76c219_e0ad_4b46_ab84_19137ac2d81e.slice/crio-c34c50821829a1193f8dfb49e515ce15d2ab5a83d7c8f8348a80c86764075bad WatchSource:0}: Error finding container c34c50821829a1193f8dfb49e515ce15d2ab5a83d7c8f8348a80c86764075bad: Status 404 returned error can't find the container with id c34c50821829a1193f8dfb49e515ce15d2ab5a83d7c8f8348a80c86764075bad
Apr 17 16:33:55.440566 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.440538    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-574d8f6f84-q4xkf"]
Apr 17 16:33:55.444082 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:33:55.444051    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0783a18_1157_4c0a_83cf_e1c50ed9d2f7.slice/crio-afcfe9376eb1cea40d0997939dd1ec2b4a517eb450572eadd4d5e2aeebdd0a6d WatchSource:0}: Error finding container afcfe9376eb1cea40d0997939dd1ec2b4a517eb450572eadd4d5e2aeebdd0a6d: Status 404 returned error can't find the container with id afcfe9376eb1cea40d0997939dd1ec2b4a517eb450572eadd4d5e2aeebdd0a6d
Apr 17 16:33:55.918229 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.918191    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8sph" event={"ID":"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d","Type":"ContainerStarted","Data":"0222dd47ac045e28382b0a071b05264c30ccaf92c71e15cb13acfd44fefd1505"}
Apr 17 16:33:55.918229 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.918231    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8sph" event={"ID":"2e7b194d-150a-4fd7-9f3d-e8475c1cd65d","Type":"ContainerStarted","Data":"6134e854ba1febc46dab04dd9558080302c5ee2934e7950651aa603c1a8a0d65"}
Apr 17 16:33:55.918433 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.918307    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f8sph"
Apr 17 16:33:55.919743 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.919716    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" event={"ID":"bc76c219-e0ad-4b46-ab84-19137ac2d81e","Type":"ContainerStarted","Data":"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"}
Apr 17 16:33:55.919886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.919747    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" event={"ID":"bc76c219-e0ad-4b46-ab84-19137ac2d81e","Type":"ContainerStarted","Data":"c34c50821829a1193f8dfb49e515ce15d2ab5a83d7c8f8348a80c86764075bad"}
Apr 17 16:33:55.919886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.919819    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:33:55.921168 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.921144    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-574d8f6f84-q4xkf" event={"ID":"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7","Type":"ContainerStarted","Data":"320b32b5559fbe92aaf026c4876075689ea8171c0127887d80feea90b3ae3b24"}
Apr 17 16:33:55.921283 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.921174    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-574d8f6f84-q4xkf" event={"ID":"e0783a18-1157-4c0a-83cf-e1c50ed9d2f7","Type":"ContainerStarted","Data":"afcfe9376eb1cea40d0997939dd1ec2b4a517eb450572eadd4d5e2aeebdd0a6d"}
Apr 17 16:33:55.934745 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.934696    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-f8sph" podStartSLOduration=129.587130322 podStartE2EDuration="2m10.934684575s" podCreationTimestamp="2026-04-17 16:31:45 +0000 UTC" firstStartedPulling="2026-04-17 16:33:53.54893869 +0000 UTC m=+160.789597103" lastFinishedPulling="2026-04-17 16:33:54.896492939 +0000 UTC m=+162.137151356" observedRunningTime="2026-04-17 16:33:55.934126575 +0000 UTC m=+163.174785010" watchObservedRunningTime="2026-04-17 16:33:55.934684575 +0000 UTC m=+163.175343005"
Apr 17 16:33:55.952040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.951976    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" podStartSLOduration=16.951962393 podStartE2EDuration="16.951962393s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:33:55.950466848 +0000 UTC m=+163.191125284" watchObservedRunningTime="2026-04-17 16:33:55.951962393 +0000 UTC m=+163.192620830"
Apr 17 16:33:55.967519 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:55.967485    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-574d8f6f84-q4xkf" podStartSLOduration=16.967473047 podStartE2EDuration="16.967473047s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:33:55.966639303 +0000 UTC m=+163.207297753" watchObservedRunningTime="2026-04-17 16:33:55.967473047 +0000 UTC m=+163.208131483"
Apr 17 16:33:56.296435 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:56.296342    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:56.299108 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:56.299082    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:56.925125 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:56.925086    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:33:56.926273 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:33:56.926249    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-574d8f6f84-q4xkf"
Apr 17 16:34:00.384492 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:00.384451    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:34:00.936813 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:00.936782    2579 generic.go:358] "Generic (PLEG): container finished" podID="d30abe04-de55-4600-af4f-e8aaf3512470" containerID="1c706595909a4882b5f377035d4bd479fdd27300818f493d35d29c78b2c9b994" exitCode=255
Apr 17 16:34:00.936967 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:00.936854    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69" event={"ID":"d30abe04-de55-4600-af4f-e8aaf3512470","Type":"ContainerDied","Data":"1c706595909a4882b5f377035d4bd479fdd27300818f493d35d29c78b2c9b994"}
Apr 17 16:34:00.943323 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:00.943303    2579 scope.go:117] "RemoveContainer" containerID="1c706595909a4882b5f377035d4bd479fdd27300818f493d35d29c78b2c9b994"
Apr 17 16:34:01.385040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:01.385002    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:34:01.940779 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:01.940741    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-667468957f-qkl69" event={"ID":"d30abe04-de55-4600-af4f-e8aaf3512470","Type":"ContainerStarted","Data":"508b5cf0b2700c8b89f716340d3fe25ecee566bd7fd1d55f590b991ad0ded1df"}
Apr 17 16:34:03.386172 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:03.386142    2579 scope.go:117] "RemoveContainer" containerID="8bb3df274349d4b5b3dae30ce2829c1036861499f98f5b84e37150f19035f4d6"
Apr 17 16:34:03.949591 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:03.949560    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:34:03.949782 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:03.949639    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5" event={"ID":"57050c6f-fb02-47e6-96b0-58ebe611a2bb","Type":"ContainerStarted","Data":"19b911caac44d1897b97c4e40fec7f8a8e9636b39024679ca69b5852d1388ace"}
Apr 17 16:34:03.950024 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:03.950005    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:34:03.954864 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:03.954834    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-9d4b6777b-vnhv5"
Apr 17 16:34:05.927196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:05.927167    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f8sph"
Apr 17 16:34:09.238280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.238246    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:34:09.259527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.259501    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dbw4n"]
Apr 17 16:34:09.264177 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.264155    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.266872 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.266849    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 17 16:34:09.266987 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.266857    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 17 16:34:09.267223 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.267205    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9k6s\""
Apr 17 16:34:09.278555 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.278528    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dbw4n"]
Apr 17 16:34:09.364722 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.364684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3515d44-4ec5-4088-a0d5-d79c4d31314b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.364722 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.364723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.364953 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.364800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3515d44-4ec5-4088-a0d5-d79c4d31314b-crio-socket\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.364953 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.364865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3515d44-4ec5-4088-a0d5-d79c4d31314b-data-volume\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.364953 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.364936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsgg\" (UniqueName: \"kubernetes.io/projected/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-api-access-lwsgg\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465611 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465575    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsgg\" (UniqueName: \"kubernetes.io/projected/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-api-access-lwsgg\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465611 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465616    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3515d44-4ec5-4088-a0d5-d79c4d31314b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465639    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465665    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3515d44-4ec5-4088-a0d5-d79c4d31314b-crio-socket\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465714    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3515d44-4ec5-4088-a0d5-d79c4d31314b-data-volume\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.465920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.465840    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3515d44-4ec5-4088-a0d5-d79c4d31314b-crio-socket\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.466130 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.466057    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3515d44-4ec5-4088-a0d5-d79c4d31314b-data-volume\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.466325 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.466308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.467973 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.467952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3515d44-4ec5-4088-a0d5-d79c4d31314b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.477580 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.477558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsgg\" (UniqueName: \"kubernetes.io/projected/a3515d44-4ec5-4088-a0d5-d79c4d31314b-kube-api-access-lwsgg\") pod \"insights-runtime-extractor-dbw4n\" (UID: \"a3515d44-4ec5-4088-a0d5-d79c4d31314b\") " pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.573046 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.572951    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dbw4n"
Apr 17 16:34:09.698369 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.698337    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dbw4n"]
Apr 17 16:34:09.701727 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:09.701702    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3515d44_4ec5_4088_a0d5_d79c4d31314b.slice/crio-f07e9bdff5033dcb776e54f991f6d18a8f2419fb1023644ec2996f842a3470a2 WatchSource:0}: Error finding container f07e9bdff5033dcb776e54f991f6d18a8f2419fb1023644ec2996f842a3470a2: Status 404 returned error can't find the container with id f07e9bdff5033dcb776e54f991f6d18a8f2419fb1023644ec2996f842a3470a2
Apr 17 16:34:09.966104 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.966068    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbw4n" event={"ID":"a3515d44-4ec5-4088-a0d5-d79c4d31314b","Type":"ContainerStarted","Data":"134009c130a76ef52a4336b443593ea854081eb44d80145da8f1c03baf494156"}
Apr 17 16:34:09.966104 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:09.966104    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbw4n" event={"ID":"a3515d44-4ec5-4088-a0d5-d79c4d31314b","Type":"ContainerStarted","Data":"f07e9bdff5033dcb776e54f991f6d18a8f2419fb1023644ec2996f842a3470a2"}
Apr 17 16:34:10.970397 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:10.970356    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbw4n" event={"ID":"a3515d44-4ec5-4088-a0d5-d79c4d31314b","Type":"ContainerStarted","Data":"1f020446c06b698c3f6c7b332a6f33485edec02203e51134f7fc348096384a4b"}
Apr 17 16:34:11.183328 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:11.183286    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:34:11.185968 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:11.185947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dfe89383-3a58-4c17-847d-fe8456a068e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-75587bd455-mk5cl\" (UID: \"dfe89383-3a58-4c17-847d-fe8456a068e8\") " pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:34:11.398844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:11.398812    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"
Apr 17 16:34:11.968009 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:11.967979    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl"]
Apr 17 16:34:11.971184 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:11.971152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe89383_3a58_4c17_847d_fe8456a068e8.slice/crio-5f08a287247df84818cdca87f4f72b9f0740675b6a50428f7374758747c0a1dc WatchSource:0}: Error finding container 5f08a287247df84818cdca87f4f72b9f0740675b6a50428f7374758747c0a1dc: Status 404 returned error can't find the container with id 5f08a287247df84818cdca87f4f72b9f0740675b6a50428f7374758747c0a1dc
Apr 17 16:34:12.980404 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:12.980364    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl" event={"ID":"dfe89383-3a58-4c17-847d-fe8456a068e8","Type":"ContainerStarted","Data":"5f08a287247df84818cdca87f4f72b9f0740675b6a50428f7374758747c0a1dc"}
Apr 17 16:34:12.982501 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:12.982470    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dbw4n" event={"ID":"a3515d44-4ec5-4088-a0d5-d79c4d31314b","Type":"ContainerStarted","Data":"bfd26087570dac3798963edc1546075e69cf6f465495d6b2af9a11a11e5151d0"}
Apr 17 16:34:13.001041 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:13.000991    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dbw4n" podStartSLOduration=1.546270832 podStartE2EDuration="4.000977293s" podCreationTimestamp="2026-04-17 16:34:09 +0000 UTC" firstStartedPulling="2026-04-17 16:34:09.755034527 +0000 UTC m=+176.995692941" lastFinishedPulling="2026-04-17 16:34:12.209740984 +0000 UTC m=+179.450399402" observedRunningTime="2026-04-17 16:34:12.999475292 +0000 UTC m=+180.240133729" watchObservedRunningTime="2026-04-17 16:34:13.000977293 +0000 UTC m=+180.241635728"
Apr 17 16:34:13.986354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:13.986320    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl" event={"ID":"dfe89383-3a58-4c17-847d-fe8456a068e8","Type":"ContainerStarted","Data":"45536a824fee0916c1e79d3d92faa5cb63d8df00744df2178d643c2122b0a9bf"}
Apr 17 16:34:14.002412 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:14.002362    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-75587bd455-mk5cl" podStartSLOduration=33.057014973 podStartE2EDuration="35.002347644s" podCreationTimestamp="2026-04-17 16:33:39 +0000 UTC" firstStartedPulling="2026-04-17 16:34:11.973321731 +0000 UTC m=+179.213980145" lastFinishedPulling="2026-04-17 16:34:13.918654399 +0000 UTC m=+181.159312816" observedRunningTime="2026-04-17 16:34:14.002143098 +0000 UTC m=+181.242801535" watchObservedRunningTime="2026-04-17 16:34:14.002347644 +0000 UTC m=+181.243006080"
Apr 17 16:34:19.243881 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:19.243849    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:34:23.963627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.963589    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mjb5z"]
Apr 17 16:34:23.967311 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.967288    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:23.970009 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.969988    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 17 16:34:23.970296 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.970276    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 17 16:34:23.970626 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.969994    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 17 16:34:23.971012 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.970994    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 17 16:34:23.971221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:23.971205    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hr7mt\""
Apr 17 16:34:24.090598 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090807 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522nj\" (UniqueName: \"kubernetes.io/projected/463d513e-b523-4a9b-90d8-857aad9bc41a-kube-api-access-522nj\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090807 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090807 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-wtmp\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-root\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-textfile\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-sys\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.090952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.091077 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.090956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-metrics-client-ca\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192038 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192000    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-wtmp\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192077    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-root\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192105    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-textfile\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192131    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-sys\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192157    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192188    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-metrics-client-ca\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192221 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192203    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-root\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192232    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192256    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-wtmp\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192284    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-522nj\" (UniqueName: \"kubernetes.io/projected/463d513e-b523-4a9b-90d8-857aad9bc41a-kube-api-access-522nj\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192338    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:24.192370    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:24.192442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls podName:463d513e-b523-4a9b-90d8-857aad9bc41a nodeName:}" failed. No retries permitted until 2026-04-17 16:34:24.69242248 +0000 UTC m=+191.933080900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls") pod "node-exporter-mjb5z" (UID: "463d513e-b523-4a9b-90d8-857aad9bc41a") : secret "node-exporter-tls" not found
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192467    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-textfile\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/463d513e-b523-4a9b-90d8-857aad9bc41a-sys\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.192949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.192779    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.193026 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.193003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d513e-b523-4a9b-90d8-857aad9bc41a-metrics-client-ca\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.194751 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.194723    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.220407 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.220337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522nj\" (UniqueName: \"kubernetes.io/projected/463d513e-b523-4a9b-90d8-857aad9bc41a-kube-api-access-522nj\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.698909 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.698878    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:24.699079 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:24.698989    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 17 16:34:24.699079 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:24.699045    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls podName:463d513e-b523-4a9b-90d8-857aad9bc41a nodeName:}" failed. No retries permitted until 2026-04-17 16:34:25.699027687 +0000 UTC m=+192.939686122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls") pod "node-exporter-mjb5z" (UID: "463d513e-b523-4a9b-90d8-857aad9bc41a") : secret "node-exporter-tls" not found
Apr 17 16:34:24.951395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.951305    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 17 16:34:24.954613 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.954596    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:24.957610 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.957586    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 17 16:34:24.957727 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.957666    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 17 16:34:24.958198 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958178    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 17 16:34:24.958423 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958385    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 17 16:34:24.958519 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958454    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 17 16:34:24.958519 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958483    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 17 16:34:24.958637 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958561    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 17 16:34:24.958717 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958700    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 17 16:34:24.958789 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958723    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sn299\""
Apr 17 16:34:24.958970 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.958955    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 17 16:34:24.970743 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:24.970720    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 17 16:34:25.001571 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.001741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.001741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.001741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.001741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.001741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-web-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-out\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.001986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vfh\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-kube-api-access-v9vfh\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.002161 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.002021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103015 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.102983    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103015 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103019    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103040    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103087    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103120    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:25.103179    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle podName:e62bcf42-1729-4ba3-82c5-bbdbac97362f nodeName:}" failed. No retries permitted until 2026-04-17 16:34:25.603152947 +0000 UTC m=+192.843811385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e62bcf42-1729-4ba3-82c5-bbdbac97362f") : configmap references non-existent config key: ca-bundle.crt
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103181    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103237    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103598 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103280    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103598 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103545    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-web-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103598 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103590    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-out\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103749 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103624    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vfh\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-kube-api-access-v9vfh\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103749 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103660    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.103749 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.103702    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.104234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.104209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.104631 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.104612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.106231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.106190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.106670 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.106642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.107258 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.107231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.107365 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.107287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.107741 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.107719    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.107985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.107916    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.108081 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.108060    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-out\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.108280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.108259    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.108475 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.108460    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e62bcf42-1729-4ba3-82c5-bbdbac97362f-web-config\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.111570 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.111543    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vfh\" (UniqueName: \"kubernetes.io/projected/e62bcf42-1729-4ba3-82c5-bbdbac97362f-kube-api-access-v9vfh\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.608069 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.608031    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.608802 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.608783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62bcf42-1729-4ba3-82c5-bbdbac97362f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e62bcf42-1729-4ba3-82c5-bbdbac97362f\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:25.709414 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.709369    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:25.711975 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.711947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/463d513e-b523-4a9b-90d8-857aad9bc41a-node-exporter-tls\") pod \"node-exporter-mjb5z\" (UID: \"463d513e-b523-4a9b-90d8-857aad9bc41a\") " pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:25.778674 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.778633    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjb5z"
Apr 17 16:34:25.788175 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:25.788146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463d513e_b523_4a9b_90d8_857aad9bc41a.slice/crio-061705ffaa054763f88001d6d8c91e4f967aaa4b6d4e97c59acdb4f54c8dccd7 WatchSource:0}: Error finding container 061705ffaa054763f88001d6d8c91e4f967aaa4b6d4e97c59acdb4f54c8dccd7: Status 404 returned error can't find the container with id 061705ffaa054763f88001d6d8c91e4f967aaa4b6d4e97c59acdb4f54c8dccd7
Apr 17 16:34:25.865402 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:25.865363    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 17 16:34:26.017113 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:26.016091    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 17 16:34:26.019748 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:26.019719    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5z" event={"ID":"463d513e-b523-4a9b-90d8-857aad9bc41a","Type":"ContainerStarted","Data":"061705ffaa054763f88001d6d8c91e4f967aaa4b6d4e97c59acdb4f54c8dccd7"}
Apr 17 16:34:26.021211 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:26.021185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62bcf42_1729_4ba3_82c5_bbdbac97362f.slice/crio-e0ed7844442c65fe68818627e61562e5bc0fe57afde4abb196be75988317aeb8 WatchSource:0}: Error finding container e0ed7844442c65fe68818627e61562e5bc0fe57afde4abb196be75988317aeb8: Status 404 returned error can't find the container with id e0ed7844442c65fe68818627e61562e5bc0fe57afde4abb196be75988317aeb8
Apr 17 16:34:27.023807 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:27.023750    2579 generic.go:358] "Generic (PLEG): container finished" podID="463d513e-b523-4a9b-90d8-857aad9bc41a" containerID="013827eb6f24368df943f7936b761dc46dfc8ea0ca84465c40a64063d7f55139" exitCode=0
Apr 17 16:34:27.024366 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:27.023799    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5z" event={"ID":"463d513e-b523-4a9b-90d8-857aad9bc41a","Type":"ContainerDied","Data":"013827eb6f24368df943f7936b761dc46dfc8ea0ca84465c40a64063d7f55139"}
Apr 17 16:34:27.024983 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:27.024961    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"e0ed7844442c65fe68818627e61562e5bc0fe57afde4abb196be75988317aeb8"}
Apr 17 16:34:28.029145 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.029108    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5z" event={"ID":"463d513e-b523-4a9b-90d8-857aad9bc41a","Type":"ContainerStarted","Data":"70370857b71b7c1cf38e17db89179370049135ca84521e6f6dae1a44eba3e80a"}
Apr 17 16:34:28.029145 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.029152    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5z" event={"ID":"463d513e-b523-4a9b-90d8-857aad9bc41a","Type":"ContainerStarted","Data":"17399fcc82dc4c48a53ac92f6371fca5619a472ba36c95a3333f1783c0fc9928"}
Apr 17 16:34:28.030395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.030369    2579 generic.go:358] "Generic (PLEG): container finished" podID="e62bcf42-1729-4ba3-82c5-bbdbac97362f" containerID="379d075ff5859be6d3e0567839cb916770a1f4737e386402b7df61d5df57b107" exitCode=0
Apr 17 16:34:28.030512 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.030441    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerDied","Data":"379d075ff5859be6d3e0567839cb916770a1f4737e386402b7df61d5df57b107"}
Apr 17 16:34:28.053912 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.053856    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mjb5z" podStartSLOduration=4.142268482 podStartE2EDuration="5.053842072s" podCreationTimestamp="2026-04-17 16:34:23 +0000 UTC" firstStartedPulling="2026-04-17 16:34:25.790133369 +0000 UTC m=+193.030791784" lastFinishedPulling="2026-04-17 16:34:26.701706951 +0000 UTC m=+193.942365374" observedRunningTime="2026-04-17 16:34:28.053590781 +0000 UTC m=+195.294249218" watchObservedRunningTime="2026-04-17 16:34:28.053842072 +0000 UTC m=+195.294500576"
Apr 17 16:34:28.494717 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.494677    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57448c675f-2zlhb"]
Apr 17 16:34:28.497396 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.497376    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.502049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.502023    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Apr 17 16:34:28.502049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.502040    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 17 16:34:28.502246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.502091    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9h2f7\""
Apr 17 16:34:28.502390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.502364    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Apr 17 16:34:28.503005 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.502988    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eurmsecaf9frj\""
Apr 17 16:34:28.503447 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.503419    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Apr 17 16:34:28.514970 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.514942    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57448c675f-2zlhb"]
Apr 17 16:34:28.534307 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-tls\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534448 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-metrics-server-audit-profiles\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534532 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-client-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534586 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-client-certs\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6e7688a4-affd-42fe-8b7d-703f4dba7b49-audit-log\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.534727 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.534650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsngq\" (UniqueName: \"kubernetes.io/projected/6e7688a4-affd-42fe-8b7d-703f4dba7b49-kube-api-access-qsngq\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636102 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636065    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6e7688a4-affd-42fe-8b7d-703f4dba7b49-audit-log\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636102 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636105    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qsngq\" (UniqueName: \"kubernetes.io/projected/6e7688a4-affd-42fe-8b7d-703f4dba7b49-kube-api-access-qsngq\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636337 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636204    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-tls\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636337 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636234    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-metrics-server-audit-profiles\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636337 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636295    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-client-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636481 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636343    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-client-certs\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636481 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636393    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.636593 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.636484    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6e7688a4-affd-42fe-8b7d-703f4dba7b49-audit-log\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.637223 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.637191    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.637481 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.637460    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6e7688a4-affd-42fe-8b7d-703f4dba7b49-metrics-server-audit-profiles\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.639144 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.639114    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-client-certs\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.639256 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.639236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-client-ca-bundle\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.639585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.639549    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6e7688a4-affd-42fe-8b7d-703f4dba7b49-secret-metrics-server-tls\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.659928 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.659861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsngq\" (UniqueName: \"kubernetes.io/projected/6e7688a4-affd-42fe-8b7d-703f4dba7b49-kube-api-access-qsngq\") pod \"metrics-server-57448c675f-2zlhb\" (UID: \"6e7688a4-affd-42fe-8b7d-703f4dba7b49\") " pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.808628 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.808465    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:28.965678 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:28.965599    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57448c675f-2zlhb"]
Apr 17 16:34:29.114007 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.113971    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"]
Apr 17 16:34:29.117688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.117665    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.120280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120254    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Apr 17 16:34:29.120427 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120254    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Apr 17 16:34:29.120697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120540    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2242b\""
Apr 17 16:34:29.120697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120580    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Apr 17 16:34:29.120697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120685    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Apr 17 16:34:29.120939 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.120804    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Apr 17 16:34:29.127501 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.127481    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Apr 17 16:34:29.132011 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.131969    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"]
Apr 17 16:34:29.141554 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.141665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.141748 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.141931 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.142016 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.142016 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.141980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-federate-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.142016 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.142007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tv6\" (UniqueName: \"kubernetes.io/projected/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-kube-api-access-99tv6\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.142162 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.142123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-metrics-client-ca\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.201172 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:29.201137    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7688a4_affd_42fe_8b7d_703f4dba7b49.slice/crio-384b9b5c8ce8b13ae155239f4843b93b85f8ca57deaa7fb067e569f0cb37dd74 WatchSource:0}: Error finding container 384b9b5c8ce8b13ae155239f4843b93b85f8ca57deaa7fb067e569f0cb37dd74: Status 404 returned error can't find the container with id 384b9b5c8ce8b13ae155239f4843b93b85f8ca57deaa7fb067e569f0cb37dd74
Apr 17 16:34:29.242805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.242780    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.242956 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.242841    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.242956 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.242873    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-federate-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.242956 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.242907    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99tv6\" (UniqueName: \"kubernetes.io/projected/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-kube-api-access-99tv6\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.243266 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.242958    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-metrics-client-ca\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.243266 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243001    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.243266 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243030    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.243266 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243088    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.244095 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.244614 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.245805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.243950    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-metrics-client-ca\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.246359 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.246332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.246769 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.246724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.246859 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.246844    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-federate-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.247301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.247282    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-telemeter-client-tls\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.252350 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.252330    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tv6\" (UniqueName: \"kubernetes.io/projected/089a8be7-7f3d-4d59-922d-88bcd46ee5a4-kube-api-access-99tv6\") pod \"telemeter-client-6f7bb84bdc-5k575\" (UID: \"089a8be7-7f3d-4d59-922d-88bcd46ee5a4\") " pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.430401 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.430374    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"
Apr 17 16:34:29.591451 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:29.591383    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7bb84bdc-5k575"]
Apr 17 16:34:29.595382 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:29.595354    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089a8be7_7f3d_4d59_922d_88bcd46ee5a4.slice/crio-65d73a694eea34ece142c7eab4b92ff73ed68e428a0aac7196bf9e5dcd69c01d WatchSource:0}: Error finding container 65d73a694eea34ece142c7eab4b92ff73ed68e428a0aac7196bf9e5dcd69c01d: Status 404 returned error can't find the container with id 65d73a694eea34ece142c7eab4b92ff73ed68e428a0aac7196bf9e5dcd69c01d
Apr 17 16:34:30.037707 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.037656    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575" event={"ID":"089a8be7-7f3d-4d59-922d-88bcd46ee5a4","Type":"ContainerStarted","Data":"65d73a694eea34ece142c7eab4b92ff73ed68e428a0aac7196bf9e5dcd69c01d"}
Apr 17 16:34:30.041276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.041177    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"801e90fcd9f2594fcf8e8130a526dadc09318b38efac9e8adc78b532a357bfc2"}
Apr 17 16:34:30.041276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.041212    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"0219f4f83a377802a6c572cd1fe762c047fa7414b51023495300ef86abfc2cd5"}
Apr 17 16:34:30.041276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.041227    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"7d02a0e82aacdc7f906107c74c23f9d19f7f6a8a2529dcfd638154448542c62b"}
Apr 17 16:34:30.041276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.041241    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"5cee1366807102b5daa69ef7ff3526d276275bd833970ae12ab84f0859ce2e0c"}
Apr 17 16:34:30.041276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.041255    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"b53013fd17647c81a2fdf52b96aaf644813eb8ad39ec3946ea38a4f6b8ee1309"}
Apr 17 16:34:30.042407 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:30.042371    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb" event={"ID":"6e7688a4-affd-42fe-8b7d-703f4dba7b49","Type":"ContainerStarted","Data":"384b9b5c8ce8b13ae155239f4843b93b85f8ca57deaa7fb067e569f0cb37dd74"}
Apr 17 16:34:31.048548 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:31.048513    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e62bcf42-1729-4ba3-82c5-bbdbac97362f","Type":"ContainerStarted","Data":"84ba3237b09a5a8563174000da3a50205dd788644c47e0d00786f89bf5f81c9a"}
Apr 17 16:34:31.050498 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:31.050469    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb" event={"ID":"6e7688a4-affd-42fe-8b7d-703f4dba7b49","Type":"ContainerStarted","Data":"71fd19312459b5d3ccf91c2c52ec9653c2d45896f3d411f5e70db3cc925de6c8"}
Apr 17 16:34:31.079787 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:31.079705    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.509571313 podStartE2EDuration="7.079690065s" podCreationTimestamp="2026-04-17 16:34:24 +0000 UTC" firstStartedPulling="2026-04-17 16:34:26.022979266 +0000 UTC m=+193.263637681" lastFinishedPulling="2026-04-17 16:34:30.593098016 +0000 UTC m=+197.833756433" observedRunningTime="2026-04-17 16:34:31.077991477 +0000 UTC m=+198.318649945" watchObservedRunningTime="2026-04-17 16:34:31.079690065 +0000 UTC m=+198.320348500"
Apr 17 16:34:31.105795 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:31.105460    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb" podStartSLOduration=1.712380677 podStartE2EDuration="3.105439769s" podCreationTimestamp="2026-04-17 16:34:28 +0000 UTC" firstStartedPulling="2026-04-17 16:34:29.203015843 +0000 UTC m=+196.443674259" lastFinishedPulling="2026-04-17 16:34:30.596074931 +0000 UTC m=+197.836733351" observedRunningTime="2026-04-17 16:34:31.104022528 +0000 UTC m=+198.344680967" watchObservedRunningTime="2026-04-17 16:34:31.105439769 +0000 UTC m=+198.346098206"
Apr 17 16:34:32.055868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:32.055821    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575" event={"ID":"089a8be7-7f3d-4d59-922d-88bcd46ee5a4","Type":"ContainerStarted","Data":"2966921be36f6de84d0dc2c2ade002282a33cc38c3f20d4384063fa27e0b0723"}
Apr 17 16:34:32.055868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:32.055872    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575" event={"ID":"089a8be7-7f3d-4d59-922d-88bcd46ee5a4","Type":"ContainerStarted","Data":"551027b269d7fea7db2cf74e742f98a42c995b0c3bb174b7a7b6c9e33e51b609"}
Apr 17 16:34:32.056432 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:32.055886    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575" event={"ID":"089a8be7-7f3d-4d59-922d-88bcd46ee5a4","Type":"ContainerStarted","Data":"3f7a6619fc95a18349f8f7314be991a15abe43decb25a8fc44efad8c2f6103f5"}
Apr 17 16:34:32.078108 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:32.078053    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f7bb84bdc-5k575" podStartSLOduration=1.388529944 podStartE2EDuration="3.078036735s" podCreationTimestamp="2026-04-17 16:34:29 +0000 UTC" firstStartedPulling="2026-04-17 16:34:29.598156567 +0000 UTC m=+196.838814985" lastFinishedPulling="2026-04-17 16:34:31.287663348 +0000 UTC m=+198.528321776" observedRunningTime="2026-04-17 16:34:32.076008872 +0000 UTC m=+199.316667307" watchObservedRunningTime="2026-04-17 16:34:32.078036735 +0000 UTC m=+199.318695170"
Apr 17 16:34:33.801809 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.801772    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:34:33.804338 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.804317    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.806710 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.806687    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5jmp\""
Apr 17 16:34:33.806926 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.806911    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Apr 17 16:34:33.807440 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.807419    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 17 16:34:33.807501 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.807464    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Apr 17 16:34:33.807811 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.807786    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 17 16:34:33.808077 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.807977    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Apr 17 16:34:33.808189 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.808164    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Apr 17 16:34:33.808325 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.808255    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Apr 17 16:34:33.812605 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.812574    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Apr 17 16:34:33.813655 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.813637    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:34:33.888280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pw7\" (UniqueName: \"kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888687 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.888687 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.888562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989702 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989666    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989712    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pw7\" (UniqueName: \"kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989739    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989795    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989813    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989851    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.989894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.989884    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.990526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.990495    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.990624 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.990525    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.990624 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.990529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.990729 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.990668    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.992264 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.992242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:33.992386 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:33.992369    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:34.008052 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.008028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pw7\" (UniqueName: \"kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7\") pod \"console-5d6c85f954-66jkq\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") " pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:34.115088 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.115055    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:34.234355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.234328    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:34:34.236573 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:34:34.236548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e2f195_aadb_4b98_a373_30120a0e9389.slice/crio-1dfd092980ef2588bdbf18d1ff64feadf880301679b9c59fc66353ab8aebbdd5 WatchSource:0}: Error finding container 1dfd092980ef2588bdbf18d1ff64feadf880301679b9c59fc66353ab8aebbdd5: Status 404 returned error can't find the container with id 1dfd092980ef2588bdbf18d1ff64feadf880301679b9c59fc66353ab8aebbdd5
Apr 17 16:34:34.256921 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.256890    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" podUID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" containerName="registry" containerID="cri-o://1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317" gracePeriod=30
Apr 17 16:34:34.491170 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.491146    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:34:34.594872 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.594838    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595067 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.594899    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595067 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.594933    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595182 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595098    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595182 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595160    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595199    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595223    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595269    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rgpw\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw\") pod \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\" (UID: \"bc76c219-e0ad-4b46-ab84-19137ac2d81e\") "
Apr 17 16:34:34.595515 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595397    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:34:34.595624 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595605    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-certificates\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.595725 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.595696    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:34:34.597356 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.597326    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:34:34.597462 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.597364    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:34:34.597550 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.597533    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:34:34.597594 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.597553    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:34:34.597865 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.597846    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw" (OuterVolumeSpecName: "kube-api-access-2rgpw") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "kube-api-access-2rgpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:34:34.603894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.603871    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bc76c219-e0ad-4b46-ab84-19137ac2d81e" (UID: "bc76c219-e0ad-4b46-ab84-19137ac2d81e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:34:34.696472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696390    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-installation-pull-secrets\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696428    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc76c219-e0ad-4b46-ab84-19137ac2d81e-image-registry-private-configuration\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696445    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-bound-sa-token\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696462    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc76c219-e0ad-4b46-ab84-19137ac2d81e-trusted-ca\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696477    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc76c219-e0ad-4b46-ab84-19137ac2d81e-ca-trust-extracted\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696491    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgpw\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-kube-api-access-2rgpw\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:34.696719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:34.696506    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc76c219-e0ad-4b46-ab84-19137ac2d81e-registry-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:34:35.068448 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.068405    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c85f954-66jkq" event={"ID":"45e2f195-aadb-4b98-a373-30120a0e9389","Type":"ContainerStarted","Data":"1dfd092980ef2588bdbf18d1ff64feadf880301679b9c59fc66353ab8aebbdd5"}
Apr 17 16:34:35.069718 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.069685    2579 generic.go:358] "Generic (PLEG): container finished" podID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" containerID="1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317" exitCode=0
Apr 17 16:34:35.069863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.069726    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" event={"ID":"bc76c219-e0ad-4b46-ab84-19137ac2d81e","Type":"ContainerDied","Data":"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"}
Apr 17 16:34:35.069863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.069751    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw" event={"ID":"bc76c219-e0ad-4b46-ab84-19137ac2d81e","Type":"ContainerDied","Data":"c34c50821829a1193f8dfb49e515ce15d2ab5a83d7c8f8348a80c86764075bad"}
Apr 17 16:34:35.069863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.069797    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f578588c7-6jgxw"
Apr 17 16:34:35.069863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.069795    2579 scope.go:117] "RemoveContainer" containerID="1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"
Apr 17 16:34:35.078877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.078857    2579 scope.go:117] "RemoveContainer" containerID="1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"
Apr 17 16:34:35.079160 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:34:35.079132    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317\": container with ID starting with 1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317 not found: ID does not exist" containerID="1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"
Apr 17 16:34:35.079267 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.079167    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317"} err="failed to get container status \"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317\": rpc error: code = NotFound desc = could not find container \"1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317\": container with ID starting with 1b247a393f50914317adb91daa595855ffe0719854197b3386e6da7f79d50317 not found: ID does not exist"
Apr 17 16:34:35.092794 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.092754    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:34:35.098571 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.098545    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f578588c7-6jgxw"]
Apr 17 16:34:35.390007 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:35.389970    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" path="/var/lib/kubelet/pods/bc76c219-e0ad-4b46-ab84-19137ac2d81e/volumes"
Apr 17 16:34:37.077823 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:37.077734    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c85f954-66jkq" event={"ID":"45e2f195-aadb-4b98-a373-30120a0e9389","Type":"ContainerStarted","Data":"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"}
Apr 17 16:34:37.098011 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:37.097959    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5d6c85f954-66jkq" podStartSLOduration=1.6469420380000002 podStartE2EDuration="4.097943065s" podCreationTimestamp="2026-04-17 16:34:33 +0000 UTC" firstStartedPulling="2026-04-17 16:34:34.238450062 +0000 UTC m=+201.479108476" lastFinishedPulling="2026-04-17 16:34:36.689451086 +0000 UTC m=+203.930109503" observedRunningTime="2026-04-17 16:34:37.096194945 +0000 UTC m=+204.336853397" watchObservedRunningTime="2026-04-17 16:34:37.097943065 +0000 UTC m=+204.338601501"
Apr 17 16:34:44.115410 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:44.115355    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:44.115410 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:44.115412    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:44.121196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:44.121169    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:45.104205 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:45.104179    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:34:48.810374 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:48.810340    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:48.810374 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:48.810377    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:34:54.128327 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:54.128288    2579 generic.go:358] "Generic (PLEG): container finished" podID="9884d527-3a7b-4c87-87d8-3df862c613e2" containerID="00ac58129096d6a88111544652512a8723122b3dfe0c25141e122e39cd0d933e" exitCode=0
Apr 17 16:34:54.128716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:54.128346    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h" event={"ID":"9884d527-3a7b-4c87-87d8-3df862c613e2","Type":"ContainerDied","Data":"00ac58129096d6a88111544652512a8723122b3dfe0c25141e122e39cd0d933e"}
Apr 17 16:34:54.128716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:54.128702    2579 scope.go:117] "RemoveContainer" containerID="00ac58129096d6a88111544652512a8723122b3dfe0c25141e122e39cd0d933e"
Apr 17 16:34:55.132833 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:34:55.132798    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-d6fc45fc5-2lq2h" event={"ID":"9884d527-3a7b-4c87-87d8-3df862c613e2","Type":"ContainerStarted","Data":"6763fa17b70bc0b7d99026d7b1652b5d9f79a0bfde615451ae22859b59a85629"}
Apr 17 16:35:08.815911 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:08.815831    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:35:08.820016 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:08.819988    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57448c675f-2zlhb"
Apr 17 16:35:09.178009 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:09.177975    2579 generic.go:358] "Generic (PLEG): container finished" podID="047abce8-32b2-4b6d-8f0e-2c7ce6ef6401" containerID="a1233d8fb9ce66690ad18291a0e82161271f7062dad2193bf4b229b69ec033d9" exitCode=0
Apr 17 16:35:09.178175 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:09.178047    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-585dfdc468-k28wq" event={"ID":"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401","Type":"ContainerDied","Data":"a1233d8fb9ce66690ad18291a0e82161271f7062dad2193bf4b229b69ec033d9"}
Apr 17 16:35:09.178501 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:09.178487    2579 scope.go:117] "RemoveContainer" containerID="a1233d8fb9ce66690ad18291a0e82161271f7062dad2193bf4b229b69ec033d9"
Apr 17 16:35:10.182775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:10.182727    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-585dfdc468-k28wq" event={"ID":"047abce8-32b2-4b6d-8f0e-2c7ce6ef6401","Type":"ContainerStarted","Data":"156f44ab11304aa81033b3106480cd4f1d8e13711ffaaa10a3f9c3b8406f7e29"}
Apr 17 16:35:24.132489 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:24.132453    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:35:24.134904 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:24.134880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4163dd4-e68b-4ae1-818d-81e2a6e33a15-metrics-certs\") pod \"network-metrics-daemon-fwn9m\" (UID: \"e4163dd4-e68b-4ae1-818d-81e2a6e33a15\") " pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:35:24.187999 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:24.187963    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bq7nv\""
Apr 17 16:35:24.196355 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:24.196319    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwn9m"
Apr 17 16:35:24.322128 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:24.322103    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwn9m"]
Apr 17 16:35:24.324702 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:35:24.324674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4163dd4_e68b_4ae1_818d_81e2a6e33a15.slice/crio-cd3b9ca27e666a8460f038e6a96d44e493dc52a6a2527e0db51e6ba337b520aa WatchSource:0}: Error finding container cd3b9ca27e666a8460f038e6a96d44e493dc52a6a2527e0db51e6ba337b520aa: Status 404 returned error can't find the container with id cd3b9ca27e666a8460f038e6a96d44e493dc52a6a2527e0db51e6ba337b520aa
Apr 17 16:35:25.227682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:25.227631    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwn9m" event={"ID":"e4163dd4-e68b-4ae1-818d-81e2a6e33a15","Type":"ContainerStarted","Data":"cd3b9ca27e666a8460f038e6a96d44e493dc52a6a2527e0db51e6ba337b520aa"}
Apr 17 16:35:26.232436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:26.232404    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwn9m" event={"ID":"e4163dd4-e68b-4ae1-818d-81e2a6e33a15","Type":"ContainerStarted","Data":"0138b0bdb0a3f65dbaa40a098beacd1391e8a0cafb88760775c3cb4d03eebc94"}
Apr 17 16:35:26.232436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:26.232439    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwn9m" event={"ID":"e4163dd4-e68b-4ae1-818d-81e2a6e33a15","Type":"ContainerStarted","Data":"319cf40653a9db2ac0dd04df5147070d4b0991ba3ce4ddbfed930f43ea61348b"}
Apr 17 16:35:26.252107 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:26.252012    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fwn9m" podStartSLOduration=252.367274733 podStartE2EDuration="4m13.251994559s" podCreationTimestamp="2026-04-17 16:31:13 +0000 UTC" firstStartedPulling="2026-04-17 16:35:24.3264805 +0000 UTC m=+251.567138915" lastFinishedPulling="2026-04-17 16:35:25.211200313 +0000 UTC m=+252.451858741" observedRunningTime="2026-04-17 16:35:26.250717606 +0000 UTC m=+253.491376041" watchObservedRunningTime="2026-04-17 16:35:26.251994559 +0000 UTC m=+253.492652990"
Apr 17 16:35:47.356235 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.356188    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:35:47.356837 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.356640    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" containerName="registry"
Apr 17 16:35:47.356837 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.356657    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" containerName="registry"
Apr 17 16:35:47.356837 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.356739    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc76c219-e0ad-4b46-ab84-19137ac2d81e" containerName="registry"
Apr 17 16:35:47.361220 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.361193    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.369849 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.369739    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:35:47.538047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwnv\" (UniqueName: \"kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538271 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538271 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538271 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538271 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.538394 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.538289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639672    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639712    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639733    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639874    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwnv\" (UniqueName: \"kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639901    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639933    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.639985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.639961    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.640543 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.640507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.640665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.640558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.640665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.640566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.640780 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.640725    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.642934 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.642906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.643123 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.643104    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.648007 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.647981    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwnv\" (UniqueName: \"kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv\") pod \"console-8dccc7464-5qhgf\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") " pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.672960 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.672927    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:47.795575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:47.795549    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:35:47.797725 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:35:47.797683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bad65f_7f4c_487c_91a1_a6b5a456564a.slice/crio-73c7d151bde3aff6ebbfc2947ec6ec628f23a2542d67149b90bdfdf4476839a7 WatchSource:0}: Error finding container 73c7d151bde3aff6ebbfc2947ec6ec628f23a2542d67149b90bdfdf4476839a7: Status 404 returned error can't find the container with id 73c7d151bde3aff6ebbfc2947ec6ec628f23a2542d67149b90bdfdf4476839a7
Apr 17 16:35:48.299585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:48.299549    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dccc7464-5qhgf" event={"ID":"a2bad65f-7f4c-487c-91a1-a6b5a456564a","Type":"ContainerStarted","Data":"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"}
Apr 17 16:35:48.299585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:48.299588    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dccc7464-5qhgf" event={"ID":"a2bad65f-7f4c-487c-91a1-a6b5a456564a","Type":"ContainerStarted","Data":"73c7d151bde3aff6ebbfc2947ec6ec628f23a2542d67149b90bdfdf4476839a7"}
Apr 17 16:35:48.333848 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:48.333796    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8dccc7464-5qhgf" podStartSLOduration=1.333780521 podStartE2EDuration="1.333780521s" podCreationTimestamp="2026-04-17 16:35:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:35:48.332100432 +0000 UTC m=+275.572758869" watchObservedRunningTime="2026-04-17 16:35:48.333780521 +0000 UTC m=+275.574438950"
Apr 17 16:35:55.313053 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:55.313013    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:35:55.315494 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:55.315464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a-cert\") pod \"ingress-canary-pmlp4\" (UID: \"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a\") " pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:35:55.588212 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:55.588127    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txqpq\""
Apr 17 16:35:55.595803 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:55.595782    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmlp4"
Apr 17 16:35:55.922426 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:55.922396    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlp4"]
Apr 17 16:35:55.925366 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:35:55.925338    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c9df82_d83b_4bc7_8eb5_dace7b9eff4a.slice/crio-111dd5dd017fe1a728f0d5f8aa245b71e1a55ddce2d440706ec9ffd07e6aa670 WatchSource:0}: Error finding container 111dd5dd017fe1a728f0d5f8aa245b71e1a55ddce2d440706ec9ffd07e6aa670: Status 404 returned error can't find the container with id 111dd5dd017fe1a728f0d5f8aa245b71e1a55ddce2d440706ec9ffd07e6aa670
Apr 17 16:35:56.323033 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:56.322952    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmlp4" event={"ID":"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a","Type":"ContainerStarted","Data":"111dd5dd017fe1a728f0d5f8aa245b71e1a55ddce2d440706ec9ffd07e6aa670"}
Apr 17 16:35:57.673333 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:57.673302    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:57.673333 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:57.673340    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:57.677978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:57.677955    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:58.329663 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:58.329620    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmlp4" event={"ID":"d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a","Type":"ContainerStarted","Data":"d7eb86bfd9f158bcec84323153fbb9a25a45ec7ee8300aaa55ea1060fae58265"}
Apr 17 16:35:58.333427 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:58.333403    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:35:58.345167 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:58.345120    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pmlp4" podStartSLOduration=251.947042802 podStartE2EDuration="4m13.34510685s" podCreationTimestamp="2026-04-17 16:31:45 +0000 UTC" firstStartedPulling="2026-04-17 16:35:55.927157286 +0000 UTC m=+283.167815704" lastFinishedPulling="2026-04-17 16:35:57.325221328 +0000 UTC m=+284.565879752" observedRunningTime="2026-04-17 16:35:58.343015397 +0000 UTC m=+285.583673833" watchObservedRunningTime="2026-04-17 16:35:58.34510685 +0000 UTC m=+285.585765316"
Apr 17 16:35:58.384949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:35:58.384915    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:36:13.278752 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:13.278722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:36:13.279368 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:13.278897    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:36:13.282702 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:13.282678    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:36:13.282913 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:13.282892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:36:13.289425 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:13.289406    2579 kubelet.go:1628] "Image garbage collection succeeded"
Apr 17 16:36:23.404103 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.404045    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5d6c85f954-66jkq" podUID="45e2f195-aadb-4b98-a373-30120a0e9389" containerName="console" containerID="cri-o://a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527" gracePeriod=15
Apr 17 16:36:23.640279 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.640258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d6c85f954-66jkq_45e2f195-aadb-4b98-a373-30120a0e9389/console/0.log"
Apr 17 16:36:23.640395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.640317    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:36:23.762504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762413    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762456    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762493    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762803 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762528    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762803 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762573    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6pw7\" (UniqueName: \"kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762803 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762597    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.762803 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762648    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config\") pod \"45e2f195-aadb-4b98-a373-30120a0e9389\" (UID: \"45e2f195-aadb-4b98-a373-30120a0e9389\") "
Apr 17 16:36:23.763034 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762983    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:36:23.763034 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762999    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config" (OuterVolumeSpecName: "console-config") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:36:23.763154 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.762989    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca" (OuterVolumeSpecName: "service-ca") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:36:23.763154 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.763044    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:36:23.764789 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.764739    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:36:23.764789 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.764752    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7" (OuterVolumeSpecName: "kube-api-access-g6pw7") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "kube-api-access-g6pw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:36:23.764789 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.764779    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "45e2f195-aadb-4b98-a373-30120a0e9389" (UID: "45e2f195-aadb-4b98-a373-30120a0e9389"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:36:23.863301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863259    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-serving-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863297    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-service-ca\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863315    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-oauth-serving-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863328    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-console-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863345    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6pw7\" (UniqueName: \"kubernetes.io/projected/45e2f195-aadb-4b98-a373-30120a0e9389-kube-api-access-g6pw7\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863358    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2f195-aadb-4b98-a373-30120a0e9389-trusted-ca-bundle\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:23.863556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:23.863370    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45e2f195-aadb-4b98-a373-30120a0e9389-console-oauth-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:36:24.404291 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d6c85f954-66jkq_45e2f195-aadb-4b98-a373-30120a0e9389/console/0.log"
Apr 17 16:36:24.404716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404300    2579 generic.go:358] "Generic (PLEG): container finished" podID="45e2f195-aadb-4b98-a373-30120a0e9389" containerID="a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527" exitCode=2
Apr 17 16:36:24.404716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404360    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6c85f954-66jkq"
Apr 17 16:36:24.404716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404376    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c85f954-66jkq" event={"ID":"45e2f195-aadb-4b98-a373-30120a0e9389","Type":"ContainerDied","Data":"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"}
Apr 17 16:36:24.404716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404400    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6c85f954-66jkq" event={"ID":"45e2f195-aadb-4b98-a373-30120a0e9389","Type":"ContainerDied","Data":"1dfd092980ef2588bdbf18d1ff64feadf880301679b9c59fc66353ab8aebbdd5"}
Apr 17 16:36:24.404716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.404414    2579 scope.go:117] "RemoveContainer" containerID="a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"
Apr 17 16:36:24.412958 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.412800    2579 scope.go:117] "RemoveContainer" containerID="a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"
Apr 17 16:36:24.413103 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:36:24.413081    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527\": container with ID starting with a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527 not found: ID does not exist" containerID="a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"
Apr 17 16:36:24.413148 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.413112    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527"} err="failed to get container status \"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527\": rpc error: code = NotFound desc = could not find container \"a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527\": container with ID starting with a9bea7611b005bbf74f00d7c386b31a34bb5adca1dfbde2adebfb338773f4527 not found: ID does not exist"
Apr 17 16:36:24.424288 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.424268    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:36:24.427745 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:24.427723    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d6c85f954-66jkq"]
Apr 17 16:36:25.388557 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:36:25.388529    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e2f195-aadb-4b98-a373-30120a0e9389" path="/var/lib/kubelet/pods/45e2f195-aadb-4b98-a373-30120a0e9389/volumes"
Apr 17 16:38:21.149683 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.149642    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"]
Apr 17 16:38:21.150314 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.150108    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e2f195-aadb-4b98-a373-30120a0e9389" containerName="console"
Apr 17 16:38:21.150314 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.150126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e2f195-aadb-4b98-a373-30120a0e9389" containerName="console"
Apr 17 16:38:21.150314 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.150201    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="45e2f195-aadb-4b98-a373-30120a0e9389" containerName="console"
Apr 17 16:38:21.152395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.152372    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.154591 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.154563    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 17 16:38:21.154722 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.154626    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 17 16:38:21.154722 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.154658    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lp58j\""
Apr 17 16:38:21.160712 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.160688    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"]
Apr 17 16:38:21.246790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.246730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4vj\" (UniqueName: \"kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.246971 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.246800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.246971 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.246839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.348279 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.348243    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4vj\" (UniqueName: \"kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.348279 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.348283    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.348511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.348314    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.348660 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.348638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.348714 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.348664    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.356976 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.356943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4vj\" (UniqueName: \"kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.462734 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.462621    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:21.583685 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.583660    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"]
Apr 17 16:38:21.586316 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:38:21.586286    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09535c27_9a29_44cd_bcf4_1021245a9738.slice/crio-233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718 WatchSource:0}: Error finding container 233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718: Status 404 returned error can't find the container with id 233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718
Apr 17 16:38:21.588170 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.588151    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 16:38:21.744531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:21.744445    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66" event={"ID":"09535c27-9a29-44cd-bcf4-1021245a9738","Type":"ContainerStarted","Data":"233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718"}
Apr 17 16:38:26.761665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:26.761625    2579 generic.go:358] "Generic (PLEG): container finished" podID="09535c27-9a29-44cd-bcf4-1021245a9738" containerID="bd6879e8e53da523d0901a37593d992be7950204c0fe5802d58cd2f1f11574c3" exitCode=0
Apr 17 16:38:26.762180 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:26.761706    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66" event={"ID":"09535c27-9a29-44cd-bcf4-1021245a9738","Type":"ContainerDied","Data":"bd6879e8e53da523d0901a37593d992be7950204c0fe5802d58cd2f1f11574c3"}
Apr 17 16:38:28.769556 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:28.769526    2579 generic.go:358] "Generic (PLEG): container finished" podID="09535c27-9a29-44cd-bcf4-1021245a9738" containerID="36b4d21026858debe9b131a20e778398f7b473479bdd78d9efa4032a348f63b0" exitCode=0
Apr 17 16:38:28.769912 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:28.769608    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66" event={"ID":"09535c27-9a29-44cd-bcf4-1021245a9738","Type":"ContainerDied","Data":"36b4d21026858debe9b131a20e778398f7b473479bdd78d9efa4032a348f63b0"}
Apr 17 16:38:35.793391 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:35.793352    2579 generic.go:358] "Generic (PLEG): container finished" podID="09535c27-9a29-44cd-bcf4-1021245a9738" containerID="4de8ba4a65780104521ad361237ec76a358971074b7f5d21cd27d9408f6ba0e5" exitCode=0
Apr 17 16:38:35.793891 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:35.793432    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66" event={"ID":"09535c27-9a29-44cd-bcf4-1021245a9738","Type":"ContainerDied","Data":"4de8ba4a65780104521ad361237ec76a358971074b7f5d21cd27d9408f6ba0e5"}
Apr 17 16:38:36.919415 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.919390    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:36.993176 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.993143    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util\") pod \"09535c27-9a29-44cd-bcf4-1021245a9738\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") "
Apr 17 16:38:36.993382 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.993241    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz4vj\" (UniqueName: \"kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj\") pod \"09535c27-9a29-44cd-bcf4-1021245a9738\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") "
Apr 17 16:38:36.993382 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.993282    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle\") pod \"09535c27-9a29-44cd-bcf4-1021245a9738\" (UID: \"09535c27-9a29-44cd-bcf4-1021245a9738\") "
Apr 17 16:38:36.993891 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.993864    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle" (OuterVolumeSpecName: "bundle") pod "09535c27-9a29-44cd-bcf4-1021245a9738" (UID: "09535c27-9a29-44cd-bcf4-1021245a9738"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:38:36.995548 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.995518    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj" (OuterVolumeSpecName: "kube-api-access-qz4vj") pod "09535c27-9a29-44cd-bcf4-1021245a9738" (UID: "09535c27-9a29-44cd-bcf4-1021245a9738"). InnerVolumeSpecName "kube-api-access-qz4vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:38:36.998291 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:36.998262    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util" (OuterVolumeSpecName: "util") pod "09535c27-9a29-44cd-bcf4-1021245a9738" (UID: "09535c27-9a29-44cd-bcf4-1021245a9738"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:38:37.094908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.094804    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4vj\" (UniqueName: \"kubernetes.io/projected/09535c27-9a29-44cd-bcf4-1021245a9738-kube-api-access-qz4vj\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:38:37.094908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.094853    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-bundle\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:38:37.094908 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.094864    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09535c27-9a29-44cd-bcf4-1021245a9738-util\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:38:37.801511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.801421    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66" event={"ID":"09535c27-9a29-44cd-bcf4-1021245a9738","Type":"ContainerDied","Data":"233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718"}
Apr 17 16:38:37.801511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.801453    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="233f7f256924e0c3bc43efc7bd3f82cdbbf6de4ba99c775d20fe651e8769f718"
Apr 17 16:38:37.801511 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:37.801478    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cl2h66"
Apr 17 16:38:43.805677 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.805640    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"]
Apr 17 16:38:43.806122 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806069    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="util"
Apr 17 16:38:43.806122 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="util"
Apr 17 16:38:43.806122 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806116    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="pull"
Apr 17 16:38:43.806231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="pull"
Apr 17 16:38:43.806231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806137    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="extract"
Apr 17 16:38:43.806231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806146    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="extract"
Apr 17 16:38:43.806231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.806212    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="09535c27-9a29-44cd-bcf4-1021245a9738" containerName="extract"
Apr 17 16:38:43.855390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.855353    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"]
Apr 17 16:38:43.855566 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.855475    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:43.858361 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.858337    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Apr 17 16:38:43.858361 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.858337    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Apr 17 16:38:43.858557 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.858360    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Apr 17 16:38:43.858557 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.858338    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-k4xfm\""
Apr 17 16:38:43.957918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.957886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjtz\" (UniqueName: \"kubernetes.io/projected/ede0f33c-0119-404e-a7d8-17996a8d3374-kube-api-access-zvjtz\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:43.957918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:43.957920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ede0f33c-0119-404e-a7d8-17996a8d3374-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.058840 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.058717    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjtz\" (UniqueName: \"kubernetes.io/projected/ede0f33c-0119-404e-a7d8-17996a8d3374-kube-api-access-zvjtz\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.058840 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.058779    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ede0f33c-0119-404e-a7d8-17996a8d3374-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.061126 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.061101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ede0f33c-0119-404e-a7d8-17996a8d3374-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.068578 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.068552    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjtz\" (UniqueName: \"kubernetes.io/projected/ede0f33c-0119-404e-a7d8-17996a8d3374-kube-api-access-zvjtz\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4\" (UID: \"ede0f33c-0119-404e-a7d8-17996a8d3374\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.165311 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.165271    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:44.305634 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.305608    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"]
Apr 17 16:38:44.308367 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:38:44.308338    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede0f33c_0119_404e_a7d8_17996a8d3374.slice/crio-9491f8b951394a42d75bc89ae84d44d04e848540c96c53370a77c023ca41075a WatchSource:0}: Error finding container 9491f8b951394a42d75bc89ae84d44d04e848540c96c53370a77c023ca41075a: Status 404 returned error can't find the container with id 9491f8b951394a42d75bc89ae84d44d04e848540c96c53370a77c023ca41075a
Apr 17 16:38:44.823146 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:44.823117    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4" event={"ID":"ede0f33c-0119-404e-a7d8-17996a8d3374","Type":"ContainerStarted","Data":"9491f8b951394a42d75bc89ae84d44d04e848540c96c53370a77c023ca41075a"}
Apr 17 16:38:49.658163 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.658124    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"]
Apr 17 16:38:49.681365 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.681335    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"]
Apr 17 16:38:49.681510 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.681458    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.686521 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.686491    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tjkqd\""
Apr 17 16:38:49.686689 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.686503    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Apr 17 16:38:49.686913 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.686900    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Apr 17 16:38:49.805886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.805851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.806057 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.805932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqd2w\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-kube-api-access-vqd2w\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.806057 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.805977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37360a0d-f0ca-4294-b2f4-120c60f01df1-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.840056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.840024    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4" event={"ID":"ede0f33c-0119-404e-a7d8-17996a8d3374","Type":"ContainerStarted","Data":"e399587a1cf21799c561f2b963cc3c91bb49420c26ff907fd713330566ac50bc"}
Apr 17 16:38:49.840255 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.840241    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:38:49.906365 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.906329    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37360a0d-f0ca-4294-b2f4-120c60f01df1-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.906538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.906374    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.906538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.906440    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vqd2w\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-kube-api-access-vqd2w\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.906538 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:49.906532    2579 secret.go:281] references non-existent secret key: tls.crt
Apr 17 16:38:49.906667 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:49.906551    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 17 16:38:49.906667 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:49.906568    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx: references non-existent secret key: tls.crt
Apr 17 16:38:49.906667 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:49.906624    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates podName:37360a0d-f0ca-4294-b2f4-120c60f01df1 nodeName:}" failed. No retries permitted until 2026-04-17 16:38:50.40660937 +0000 UTC m=+457.647267784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates") pod "keda-metrics-apiserver-7c9f485588-j6fjx" (UID: "37360a0d-f0ca-4294-b2f4-120c60f01df1") : references non-existent secret key: tls.crt
Apr 17 16:38:49.906832 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.906811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37360a0d-f0ca-4294-b2f4-120c60f01df1-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:49.925792 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:49.925698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqd2w\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-kube-api-access-vqd2w\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:50.021647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.021582    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4" podStartSLOduration=2.548193245 podStartE2EDuration="7.02156603s" podCreationTimestamp="2026-04-17 16:38:43 +0000 UTC" firstStartedPulling="2026-04-17 16:38:44.310205412 +0000 UTC m=+451.550863827" lastFinishedPulling="2026-04-17 16:38:48.783578185 +0000 UTC m=+456.024236612" observedRunningTime="2026-04-17 16:38:49.903096638 +0000 UTC m=+457.143755073" watchObservedRunningTime="2026-04-17 16:38:50.02156603 +0000 UTC m=+457.262224465"
Apr 17 16:38:50.022381 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.022362    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-9jhj9"]
Apr 17 16:38:50.046241 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.046201    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-9jhj9"]
Apr 17 16:38:50.046381 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.046343    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.050154 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.050134    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Apr 17 16:38:50.208973 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.208878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94c5\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-kube-api-access-p94c5\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.208973 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.208941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.310310 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.310270    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p94c5\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-kube-api-access-p94c5\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.310493 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.310336    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.310493 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.310456    2579 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Apr 17 16:38:50.310493 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.310485    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-9jhj9: secret "keda-admission-webhooks-certs" not found
Apr 17 16:38:50.310599 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.310532    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates podName:3ece2d52-676e-45de-9ec0-1a79e552a899 nodeName:}" failed. No retries permitted until 2026-04-17 16:38:50.81051511 +0000 UTC m=+458.051173528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates") pod "keda-admission-cf49989db-9jhj9" (UID: "3ece2d52-676e-45de-9ec0-1a79e552a899") : secret "keda-admission-webhooks-certs" not found
Apr 17 16:38:50.319781 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.319732    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94c5\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-kube-api-access-p94c5\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.411029 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.410988    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:50.411218 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.411104    2579 secret.go:281] references non-existent secret key: tls.crt
Apr 17 16:38:50.411218 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.411116    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 17 16:38:50.411218 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.411143    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx: references non-existent secret key: tls.crt
Apr 17 16:38:50.411218 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:50.411190    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates podName:37360a0d-f0ca-4294-b2f4-120c60f01df1 nodeName:}" failed. No retries permitted until 2026-04-17 16:38:51.411177093 +0000 UTC m=+458.651835507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates") pod "keda-metrics-apiserver-7c9f485588-j6fjx" (UID: "37360a0d-f0ca-4294-b2f4-120c60f01df1") : references non-existent secret key: tls.crt
Apr 17 16:38:50.814507 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.814464    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.817206 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.817179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ece2d52-676e-45de-9ec0-1a79e552a899-certificates\") pod \"keda-admission-cf49989db-9jhj9\" (UID: \"3ece2d52-676e-45de-9ec0-1a79e552a899\") " pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:50.959648 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:50.959617    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:51.085234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:51.085206    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-9jhj9"]
Apr 17 16:38:51.087365 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:38:51.087341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ece2d52_676e_45de_9ec0_1a79e552a899.slice/crio-24252fe6d7c6526e28c303d54177cf58d2129c1275cc510f09d75b6658e78158 WatchSource:0}: Error finding container 24252fe6d7c6526e28c303d54177cf58d2129c1275cc510f09d75b6658e78158: Status 404 returned error can't find the container with id 24252fe6d7c6526e28c303d54177cf58d2129c1275cc510f09d75b6658e78158
Apr 17 16:38:51.420866 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:51.420827    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:51.421110 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:51.420932    2579 secret.go:281] references non-existent secret key: tls.crt
Apr 17 16:38:51.421110 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:51.420944    2579 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 17 16:38:51.421110 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:51.420962    2579 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx: references non-existent secret key: tls.crt
Apr 17 16:38:51.421110 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:38:51.421010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates podName:37360a0d-f0ca-4294-b2f4-120c60f01df1 nodeName:}" failed. No retries permitted until 2026-04-17 16:38:53.420997216 +0000 UTC m=+460.661655630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates") pod "keda-metrics-apiserver-7c9f485588-j6fjx" (UID: "37360a0d-f0ca-4294-b2f4-120c60f01df1") : references non-existent secret key: tls.crt
Apr 17 16:38:51.847857 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:51.847775    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-9jhj9" event={"ID":"3ece2d52-676e-45de-9ec0-1a79e552a899","Type":"ContainerStarted","Data":"24252fe6d7c6526e28c303d54177cf58d2129c1275cc510f09d75b6658e78158"}
Apr 17 16:38:52.852630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:52.852529    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-9jhj9" event={"ID":"3ece2d52-676e-45de-9ec0-1a79e552a899","Type":"ContainerStarted","Data":"833835d8120c4def18c3de54d1dc21fc3baf98ecf034e876522a508c9c1e3cfa"}
Apr 17 16:38:52.852630 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:52.852612    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:38:52.875171 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:52.875123    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-9jhj9" podStartSLOduration=2.436880563 podStartE2EDuration="3.875104061s" podCreationTimestamp="2026-04-17 16:38:49 +0000 UTC" firstStartedPulling="2026-04-17 16:38:51.08900974 +0000 UTC m=+458.329668154" lastFinishedPulling="2026-04-17 16:38:52.527233223 +0000 UTC m=+459.767891652" observedRunningTime="2026-04-17 16:38:52.873168399 +0000 UTC m=+460.113826840" watchObservedRunningTime="2026-04-17 16:38:52.875104061 +0000 UTC m=+460.115762526"
Apr 17 16:38:53.439555 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:53.439513    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:53.442224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:53.442197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37360a0d-f0ca-4294-b2f4-120c60f01df1-certificates\") pod \"keda-metrics-apiserver-7c9f485588-j6fjx\" (UID: \"37360a0d-f0ca-4294-b2f4-120c60f01df1\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:53.594090 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:53.594055    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:53.734162 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:53.734078    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"]
Apr 17 16:38:53.737112 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:38:53.737084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37360a0d_f0ca_4294_b2f4_120c60f01df1.slice/crio-0578a5ce634176a6c279df6a1060cfa1b7de11e39975947a594a4d28d781e4be WatchSource:0}: Error finding container 0578a5ce634176a6c279df6a1060cfa1b7de11e39975947a594a4d28d781e4be: Status 404 returned error can't find the container with id 0578a5ce634176a6c279df6a1060cfa1b7de11e39975947a594a4d28d781e4be
Apr 17 16:38:53.856914 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:53.856880    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx" event={"ID":"37360a0d-f0ca-4294-b2f4-120c60f01df1","Type":"ContainerStarted","Data":"0578a5ce634176a6c279df6a1060cfa1b7de11e39975947a594a4d28d781e4be"}
Apr 17 16:38:56.875119 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:56.875083    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx" event={"ID":"37360a0d-f0ca-4294-b2f4-120c60f01df1","Type":"ContainerStarted","Data":"e967ce979eaea7638a765b8d2d597d4914c7ff5acdd0181a36e55d6d4d4660c1"}
Apr 17 16:38:56.875494 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:56.875208    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:38:56.898187 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:38:56.898142    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx" podStartSLOduration=5.534546779 podStartE2EDuration="7.898127087s" podCreationTimestamp="2026-04-17 16:38:49 +0000 UTC" firstStartedPulling="2026-04-17 16:38:53.738422022 +0000 UTC m=+460.979080438" lastFinishedPulling="2026-04-17 16:38:56.102002322 +0000 UTC m=+463.342660746" observedRunningTime="2026-04-17 16:38:56.896497085 +0000 UTC m=+464.137155522" watchObservedRunningTime="2026-04-17 16:38:56.898127087 +0000 UTC m=+464.138785522"
Apr 17 16:39:07.883482 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:07.883454    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-j6fjx"
Apr 17 16:39:10.846144 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:10.846115    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-fgvv4"
Apr 17 16:39:13.859032 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:13.858999    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-9jhj9"
Apr 17 16:39:33.826091 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.826054    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8696869745-4qc52"]
Apr 17 16:39:33.829959 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.829942    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.844643 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.844612    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8696869745-4qc52"]
Apr 17 16:39:33.898096 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-oauth-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898096 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-trusted-ca-bundle\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898300 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898300 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj5t\" (UniqueName: \"kubernetes.io/projected/85efab03-3e5a-495a-a123-ef8f18806455-kube-api-access-8zj5t\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898300 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-oauth-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898401 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-console-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.898434 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.898406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-service-ca\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.998917 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.998885    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-service-ca\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999133 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.998926    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-oauth-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999133 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.999002    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-trusted-ca-bundle\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999133 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.999042    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999295 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.999159    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj5t\" (UniqueName: \"kubernetes.io/projected/85efab03-3e5a-495a-a123-ef8f18806455-kube-api-access-8zj5t\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999295 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.999221    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-oauth-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:33.999399 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:33.999301    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-console-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.000040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.000003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-oauth-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.000171 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.000090    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-trusted-ca-bundle\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.000171 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.000102    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-console-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.000251 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.000171    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85efab03-3e5a-495a-a123-ef8f18806455-service-ca\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.001950 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.001928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-oauth-config\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.002076 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.002058    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85efab03-3e5a-495a-a123-ef8f18806455-console-serving-cert\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.012466 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.012438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj5t\" (UniqueName: \"kubernetes.io/projected/85efab03-3e5a-495a-a123-ef8f18806455-kube-api-access-8zj5t\") pod \"console-8696869745-4qc52\" (UID: \"85efab03-3e5a-495a-a123-ef8f18806455\") " pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.138678 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.138640    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:34.267917 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:34.267892    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8696869745-4qc52"]
Apr 17 16:39:34.270548 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:39:34.270512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85efab03_3e5a_495a_a123_ef8f18806455.slice/crio-425020e4f7600be5b990f9195dcaec8bdd113ec1bfb84a25220973001eba7d48 WatchSource:0}: Error finding container 425020e4f7600be5b990f9195dcaec8bdd113ec1bfb84a25220973001eba7d48: Status 404 returned error can't find the container with id 425020e4f7600be5b990f9195dcaec8bdd113ec1bfb84a25220973001eba7d48
Apr 17 16:39:35.006827 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:35.006792    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8696869745-4qc52" event={"ID":"85efab03-3e5a-495a-a123-ef8f18806455","Type":"ContainerStarted","Data":"35b670f8417a4ef130712df1d6b91d393830b58a9b8d3ee053722b50070f2923"}
Apr 17 16:39:35.006827 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:35.006832    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8696869745-4qc52" event={"ID":"85efab03-3e5a-495a-a123-ef8f18806455","Type":"ContainerStarted","Data":"425020e4f7600be5b990f9195dcaec8bdd113ec1bfb84a25220973001eba7d48"}
Apr 17 16:39:35.030371 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:35.030323    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8696869745-4qc52" podStartSLOduration=2.03030722 podStartE2EDuration="2.03030722s" podCreationTimestamp="2026-04-17 16:39:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:39:35.028553708 +0000 UTC m=+502.269212144" watchObservedRunningTime="2026-04-17 16:39:35.03030722 +0000 UTC m=+502.270965657"
Apr 17 16:39:44.139309 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:44.139265    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:44.139309 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:44.139307    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:44.144069 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:44.144040    2579 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:45.042214 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:45.042187    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8696869745-4qc52"
Apr 17 16:39:45.130087 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:39:45.130046    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:40:00.512010 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.511971    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:00.515251 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.515233    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.517393 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.517371    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fclwl\""
Apr 17 16:40:00.517496 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.517371    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 17 16:40:00.518112 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.518095    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 17 16:40:00.518112 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.518108    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Apr 17 16:40:00.527207 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.527178    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:00.532992 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.532961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst58\" (UniqueName: \"kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.533113 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.533001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.634021 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.633982    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.634217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.634104    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dst58\" (UniqueName: \"kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.636402 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.636379    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.645861 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.645831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst58\" (UniqueName: \"kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58\") pod \"kserve-controller-manager-85bb65f8c4-g89pq\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") " pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.828371 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.828283    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:00.958548 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:00.958504    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:00.965971 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:40:00.965943    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26229459_68c3_4057_adcb_3fbcaaf7ca80.slice/crio-da0470c3a0c8c5387cd7bd86dbc3d17a7f99e81a9fa2edd126ab0e7af7fd3e2c WatchSource:0}: Error finding container da0470c3a0c8c5387cd7bd86dbc3d17a7f99e81a9fa2edd126ab0e7af7fd3e2c: Status 404 returned error can't find the container with id da0470c3a0c8c5387cd7bd86dbc3d17a7f99e81a9fa2edd126ab0e7af7fd3e2c
Apr 17 16:40:01.090597 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:01.090509    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" event={"ID":"26229459-68c3-4057-adcb-3fbcaaf7ca80","Type":"ContainerStarted","Data":"da0470c3a0c8c5387cd7bd86dbc3d17a7f99e81a9fa2edd126ab0e7af7fd3e2c"}
Apr 17 16:40:04.105140 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:04.105042    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" event={"ID":"26229459-68c3-4057-adcb-3fbcaaf7ca80","Type":"ContainerStarted","Data":"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"}
Apr 17 16:40:04.105538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:04.105144    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:04.122370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:04.122323    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" podStartSLOduration=1.342500098 podStartE2EDuration="4.12230806s" podCreationTimestamp="2026-04-17 16:40:00 +0000 UTC" firstStartedPulling="2026-04-17 16:40:00.967359056 +0000 UTC m=+528.208017484" lastFinishedPulling="2026-04-17 16:40:03.747167032 +0000 UTC m=+530.987825446" observedRunningTime="2026-04-17 16:40:04.12065518 +0000 UTC m=+531.361313618" watchObservedRunningTime="2026-04-17 16:40:04.12230806 +0000 UTC m=+531.362966496"
Apr 17 16:40:10.150962 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.150889    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8dccc7464-5qhgf" podUID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" containerName="console" containerID="cri-o://0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6" gracePeriod=15
Apr 17 16:40:10.393572 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.393548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8dccc7464-5qhgf_a2bad65f-7f4c-487c-91a1-a6b5a456564a/console/0.log"
Apr 17 16:40:10.393694 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.393606    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:40:10.526846 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526743    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.526846 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526821    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.526846 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526839    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.527181 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526862    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvwnv\" (UniqueName: \"kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.527181 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526881    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.527181 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526913    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.527181 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.526954    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert\") pod \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\" (UID: \"a2bad65f-7f4c-487c-91a1-a6b5a456564a\") "
Apr 17 16:40:10.527386 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.527342    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:40:10.527447 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.527374    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config" (OuterVolumeSpecName: "console-config") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:40:10.527447 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.527426    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:40:10.527679 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.527647    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca" (OuterVolumeSpecName: "service-ca") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:40:10.528981 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.528957    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv" (OuterVolumeSpecName: "kube-api-access-zvwnv") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "kube-api-access-zvwnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:40:10.529076 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.528980    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:40:10.529120 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.529107    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a2bad65f-7f4c-487c-91a1-a6b5a456564a" (UID: "a2bad65f-7f4c-487c-91a1-a6b5a456564a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:40:10.628609 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628578    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-trusted-ca-bundle\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628609 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628608    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-serving-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628618    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-oauth-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628628    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-console-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628637    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-oauth-serving-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628645    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwnv\" (UniqueName: \"kubernetes.io/projected/a2bad65f-7f4c-487c-91a1-a6b5a456564a-kube-api-access-zvwnv\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:10.628826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:10.628656    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2bad65f-7f4c-487c-91a1-a6b5a456564a-service-ca\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:11.128693 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128661    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8dccc7464-5qhgf_a2bad65f-7f4c-487c-91a1-a6b5a456564a/console/0.log"
Apr 17 16:40:11.128924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128700    2579 generic.go:358] "Generic (PLEG): container finished" podID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" containerID="0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6" exitCode=2
Apr 17 16:40:11.128924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128794    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dccc7464-5qhgf"
Apr 17 16:40:11.128924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128793    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dccc7464-5qhgf" event={"ID":"a2bad65f-7f4c-487c-91a1-a6b5a456564a","Type":"ContainerDied","Data":"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"}
Apr 17 16:40:11.128924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128833    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dccc7464-5qhgf" event={"ID":"a2bad65f-7f4c-487c-91a1-a6b5a456564a","Type":"ContainerDied","Data":"73c7d151bde3aff6ebbfc2947ec6ec628f23a2542d67149b90bdfdf4476839a7"}
Apr 17 16:40:11.128924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.128848    2579 scope.go:117] "RemoveContainer" containerID="0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"
Apr 17 16:40:11.137479 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.137462    2579 scope.go:117] "RemoveContainer" containerID="0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"
Apr 17 16:40:11.137725 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:40:11.137698    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6\": container with ID starting with 0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6 not found: ID does not exist" containerID="0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"
Apr 17 16:40:11.137806 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.137725    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6"} err="failed to get container status \"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6\": rpc error: code = NotFound desc = could not find container \"0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6\": container with ID starting with 0a781376e15ea3749f2b70dc295bd5452a3344deda7b49176e05f909d317b8d6 not found: ID does not exist"
Apr 17 16:40:11.149642 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.149617    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:40:11.153224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.153202    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8dccc7464-5qhgf"]
Apr 17 16:40:11.389418 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:11.389335    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" path="/var/lib/kubelet/pods/a2bad65f-7f4c-487c-91a1-a6b5a456564a/volumes"
Apr 17 16:40:35.113844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.113809    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:35.916424 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.916391    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:35.916606 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.916591    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" podUID="26229459-68c3-4057-adcb-3fbcaaf7ca80" containerName="manager" containerID="cri-o://4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5" gracePeriod=10
Apr 17 16:40:35.940451 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.940424    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-xr4sz"]
Apr 17 16:40:35.940748 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.940737    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" containerName="console"
Apr 17 16:40:35.940819 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.940750    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" containerName="console"
Apr 17 16:40:35.940856 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.940820    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2bad65f-7f4c-487c-91a1-a6b5a456564a" containerName="console"
Apr 17 16:40:35.943555 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.943540    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:35.950263 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:35.950237    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-xr4sz"]
Apr 17 16:40:36.040799 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.040745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdqs\" (UniqueName: \"kubernetes.io/projected/bdc5ab43-2484-4907-9fe0-3d0db28f6633-kube-api-access-fjdqs\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.040942 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.040878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc5ab43-2484-4907-9fe0-3d0db28f6633-cert\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.142243 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.142214    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdqs\" (UniqueName: \"kubernetes.io/projected/bdc5ab43-2484-4907-9fe0-3d0db28f6633-kube-api-access-fjdqs\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.142623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.142276    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc5ab43-2484-4907-9fe0-3d0db28f6633-cert\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.144575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.144557    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc5ab43-2484-4907-9fe0-3d0db28f6633-cert\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.150878 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.150861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdqs\" (UniqueName: \"kubernetes.io/projected/bdc5ab43-2484-4907-9fe0-3d0db28f6633-kube-api-access-fjdqs\") pod \"kserve-controller-manager-85bb65f8c4-xr4sz\" (UID: \"bdc5ab43-2484-4907-9fe0-3d0db28f6633\") " pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.162286 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.162265    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:36.218038 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.217955    2579 generic.go:358] "Generic (PLEG): container finished" podID="26229459-68c3-4057-adcb-3fbcaaf7ca80" containerID="4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5" exitCode=0
Apr 17 16:40:36.218202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.218038    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq"
Apr 17 16:40:36.218202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.218039    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" event={"ID":"26229459-68c3-4057-adcb-3fbcaaf7ca80","Type":"ContainerDied","Data":"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"}
Apr 17 16:40:36.218202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.218085    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-g89pq" event={"ID":"26229459-68c3-4057-adcb-3fbcaaf7ca80","Type":"ContainerDied","Data":"da0470c3a0c8c5387cd7bd86dbc3d17a7f99e81a9fa2edd126ab0e7af7fd3e2c"}
Apr 17 16:40:36.218202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.218105    2579 scope.go:117] "RemoveContainer" containerID="4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"
Apr 17 16:40:36.225996 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.225978    2579 scope.go:117] "RemoveContainer" containerID="4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"
Apr 17 16:40:36.226233 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:40:36.226213    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5\": container with ID starting with 4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5 not found: ID does not exist" containerID="4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"
Apr 17 16:40:36.226282 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.226242    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5"} err="failed to get container status \"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5\": rpc error: code = NotFound desc = could not find container \"4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5\": container with ID starting with 4fb023b22a9d66792cf8e31fd61a823280e6de51d6212cdf5a38fd03024ae4a5 not found: ID does not exist"
Apr 17 16:40:36.293485 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.293453    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:36.344111 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.344080    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dst58\" (UniqueName: \"kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58\") pod \"26229459-68c3-4057-adcb-3fbcaaf7ca80\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") "
Apr 17 16:40:36.344258 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.344131    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert\") pod \"26229459-68c3-4057-adcb-3fbcaaf7ca80\" (UID: \"26229459-68c3-4057-adcb-3fbcaaf7ca80\") "
Apr 17 16:40:36.346574 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.346546    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert" (OuterVolumeSpecName: "cert") pod "26229459-68c3-4057-adcb-3fbcaaf7ca80" (UID: "26229459-68c3-4057-adcb-3fbcaaf7ca80"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:40:36.346863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.346837    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58" (OuterVolumeSpecName: "kube-api-access-dst58") pod "26229459-68c3-4057-adcb-3fbcaaf7ca80" (UID: "26229459-68c3-4057-adcb-3fbcaaf7ca80"). InnerVolumeSpecName "kube-api-access-dst58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:40:36.414856 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.414823    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-xr4sz"]
Apr 17 16:40:36.418016 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:40:36.417988    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc5ab43_2484_4907_9fe0_3d0db28f6633.slice/crio-956fb7503d5e98edff76ce493045943c9d33de329364b50f5aa77eb4ff27315f WatchSource:0}: Error finding container 956fb7503d5e98edff76ce493045943c9d33de329364b50f5aa77eb4ff27315f: Status 404 returned error can't find the container with id 956fb7503d5e98edff76ce493045943c9d33de329364b50f5aa77eb4ff27315f
Apr 17 16:40:36.445071 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.445040    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst58\" (UniqueName: \"kubernetes.io/projected/26229459-68c3-4057-adcb-3fbcaaf7ca80-kube-api-access-dst58\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:36.445071 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.445066    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26229459-68c3-4057-adcb-3fbcaaf7ca80-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:40:36.538566 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.538531    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:36.541807 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:36.541783    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85bb65f8c4-g89pq"]
Apr 17 16:40:37.222495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:37.222460    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz" event={"ID":"bdc5ab43-2484-4907-9fe0-3d0db28f6633","Type":"ContainerStarted","Data":"bbf117f6db01f0d1f0598ccd389b6901d33a6ca32dd6ee2b61885cf258fb717a"}
Apr 17 16:40:37.222495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:37.222493    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz" event={"ID":"bdc5ab43-2484-4907-9fe0-3d0db28f6633","Type":"ContainerStarted","Data":"956fb7503d5e98edff76ce493045943c9d33de329364b50f5aa77eb4ff27315f"}
Apr 17 16:40:37.223004 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:37.222566    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:40:37.237230 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:37.237183    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz" podStartSLOduration=1.896495974 podStartE2EDuration="2.237166177s" podCreationTimestamp="2026-04-17 16:40:35 +0000 UTC" firstStartedPulling="2026-04-17 16:40:36.419314604 +0000 UTC m=+563.659973017" lastFinishedPulling="2026-04-17 16:40:36.759984807 +0000 UTC m=+564.000643220" observedRunningTime="2026-04-17 16:40:37.237081755 +0000 UTC m=+564.477740191" watchObservedRunningTime="2026-04-17 16:40:37.237166177 +0000 UTC m=+564.477824613"
Apr 17 16:40:37.388868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:40:37.388839    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26229459-68c3-4057-adcb-3fbcaaf7ca80" path="/var/lib/kubelet/pods/26229459-68c3-4057-adcb-3fbcaaf7ca80/volumes"
Apr 17 16:41:08.230388 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:08.230360    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85bb65f8c4-xr4sz"
Apr 17 16:41:09.171051 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.171014    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-dzbnm"]
Apr 17 16:41:09.171513 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.171498    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26229459-68c3-4057-adcb-3fbcaaf7ca80" containerName="manager"
Apr 17 16:41:09.171592 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.171515    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26229459-68c3-4057-adcb-3fbcaaf7ca80" containerName="manager"
Apr 17 16:41:09.171592 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.171586    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="26229459-68c3-4057-adcb-3fbcaaf7ca80" containerName="manager"
Apr 17 16:41:09.174897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.174877    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.176978 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.176958    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Apr 17 16:41:09.177200 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.177185    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-x27bl\""
Apr 17 16:41:09.183832 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.183809    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-dzbnm"]
Apr 17 16:41:09.232780 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.232732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5d9k\" (UniqueName: \"kubernetes.io/projected/37533871-b460-45f9-9a46-3e1dbf9b003c-kube-api-access-g5d9k\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.233171 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.232818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.333476 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.333425    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g5d9k\" (UniqueName: \"kubernetes.io/projected/37533871-b460-45f9-9a46-3e1dbf9b003c-kube-api-access-g5d9k\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.333693 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.333581    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.333693 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:09.333681    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Apr 17 16:41:09.333819 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:09.333732    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert podName:37533871-b460-45f9-9a46-3e1dbf9b003c nodeName:}" failed. No retries permitted until 2026-04-17 16:41:09.833716786 +0000 UTC m=+597.074375199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert") pod "odh-model-controller-696fc77849-dzbnm" (UID: "37533871-b460-45f9-9a46-3e1dbf9b003c") : secret "odh-model-controller-webhook-cert" not found
Apr 17 16:41:09.342500 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.342474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5d9k\" (UniqueName: \"kubernetes.io/projected/37533871-b460-45f9-9a46-3e1dbf9b003c-kube-api-access-g5d9k\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.838454 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.838419    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:09.840721 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:09.840702    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37533871-b460-45f9-9a46-3e1dbf9b003c-cert\") pod \"odh-model-controller-696fc77849-dzbnm\" (UID: \"37533871-b460-45f9-9a46-3e1dbf9b003c\") " pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:10.087280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:10.087240    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:10.242531 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:10.242490    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-dzbnm"]
Apr 17 16:41:10.245495 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:41:10.245469    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37533871_b460_45f9_9a46_3e1dbf9b003c.slice/crio-977117a75908ef215cb665f11eeb2670b4f294c515ed30c7516e4bc73b3b8cc1 WatchSource:0}: Error finding container 977117a75908ef215cb665f11eeb2670b4f294c515ed30c7516e4bc73b3b8cc1: Status 404 returned error can't find the container with id 977117a75908ef215cb665f11eeb2670b4f294c515ed30c7516e4bc73b3b8cc1
Apr 17 16:41:10.331955 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:10.331920    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-dzbnm" event={"ID":"37533871-b460-45f9-9a46-3e1dbf9b003c","Type":"ContainerStarted","Data":"977117a75908ef215cb665f11eeb2670b4f294c515ed30c7516e4bc73b3b8cc1"}
Apr 17 16:41:13.308911 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.308826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:41:13.309283 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.309066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:41:13.312190 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.312169    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:41:13.312390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.312376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:41:13.343602 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.343562    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-dzbnm" event={"ID":"37533871-b460-45f9-9a46-3e1dbf9b003c","Type":"ContainerStarted","Data":"e6ab4eb6c82a3e1df5de1e67f23d859a1c5e96d9185708f55190f10f0aab0905"}
Apr 17 16:41:13.343742 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.343674    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:13.361277 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:13.361232    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-dzbnm" podStartSLOduration=1.587029524 podStartE2EDuration="4.361218963s" podCreationTimestamp="2026-04-17 16:41:09 +0000 UTC" firstStartedPulling="2026-04-17 16:41:10.246813506 +0000 UTC m=+597.487471919" lastFinishedPulling="2026-04-17 16:41:13.021002941 +0000 UTC m=+600.261661358" observedRunningTime="2026-04-17 16:41:13.360337319 +0000 UTC m=+600.600995756" watchObservedRunningTime="2026-04-17 16:41:13.361218963 +0000 UTC m=+600.601877398"
Apr 17 16:41:24.349407 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:24.349375    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-dzbnm"
Apr 17 16:41:36.480202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.480126    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:36.484370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.484351    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.486575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.486556    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Apr 17 16:41:36.486625 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.486556    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mdnc6\""
Apr 17 16:41:36.490042 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.490016    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:36.575545 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.575507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkzq\" (UniqueName: \"kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.575734 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.575574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.676979 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.676933    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-klkzq\" (UniqueName: \"kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.677129 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.676994    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.677335 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.677318    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.685578 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.685552    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkzq\" (UniqueName: \"kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq\") pod \"seaweedfs-tls-custom-ddd4dbfd-fb5qt\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.794647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.794546    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:36.916890 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:36.916858    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:37.422230 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:37.422191    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" event={"ID":"406c0305-7dc6-422b-9cad-e1a97e052456","Type":"ContainerStarted","Data":"cb24d3bb61c75888e6c98774411bb4342e38f631fa82c19bcb4513e26e305386"}
Apr 17 16:41:40.434458 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:40.434422    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" event={"ID":"406c0305-7dc6-422b-9cad-e1a97e052456","Type":"ContainerStarted","Data":"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"}
Apr 17 16:41:40.452071 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:40.452022    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" podStartSLOduration=1.7468271560000002 podStartE2EDuration="4.452006767s" podCreationTimestamp="2026-04-17 16:41:36 +0000 UTC" firstStartedPulling="2026-04-17 16:41:36.923179991 +0000 UTC m=+624.163838404" lastFinishedPulling="2026-04-17 16:41:39.628359595 +0000 UTC m=+626.869018015" observedRunningTime="2026-04-17 16:41:40.450596579 +0000 UTC m=+627.691255017" watchObservedRunningTime="2026-04-17 16:41:40.452006767 +0000 UTC m=+627.692665203"
Apr 17 16:41:41.975623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:41.975584    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:42.446774 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:42.446704    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" podUID="406c0305-7dc6-422b-9cad-e1a97e052456" containerName="seaweedfs-tls-custom" containerID="cri-o://5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2" gracePeriod=30
Apr 17 16:41:43.693808 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.693785    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:43.739954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.739852    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klkzq\" (UniqueName: \"kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq\") pod \"406c0305-7dc6-422b-9cad-e1a97e052456\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") "
Apr 17 16:41:43.740116 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.739961    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data\") pod \"406c0305-7dc6-422b-9cad-e1a97e052456\" (UID: \"406c0305-7dc6-422b-9cad-e1a97e052456\") "
Apr 17 16:41:43.741237 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.741204    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data" (OuterVolumeSpecName: "data") pod "406c0305-7dc6-422b-9cad-e1a97e052456" (UID: "406c0305-7dc6-422b-9cad-e1a97e052456"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:41:43.741965 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.741935    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq" (OuterVolumeSpecName: "kube-api-access-klkzq") pod "406c0305-7dc6-422b-9cad-e1a97e052456" (UID: "406c0305-7dc6-422b-9cad-e1a97e052456"). InnerVolumeSpecName "kube-api-access-klkzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:41:43.841359 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.841323    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkzq\" (UniqueName: \"kubernetes.io/projected/406c0305-7dc6-422b-9cad-e1a97e052456-kube-api-access-klkzq\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:41:43.841359 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:43.841355    2579 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/406c0305-7dc6-422b-9cad-e1a97e052456-data\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:41:44.453886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.453850    2579 generic.go:358] "Generic (PLEG): container finished" podID="406c0305-7dc6-422b-9cad-e1a97e052456" containerID="5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2" exitCode=0
Apr 17 16:41:44.454087 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.453908    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"
Apr 17 16:41:44.454087 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.453933    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" event={"ID":"406c0305-7dc6-422b-9cad-e1a97e052456","Type":"ContainerDied","Data":"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"}
Apr 17 16:41:44.454087 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.453968    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt" event={"ID":"406c0305-7dc6-422b-9cad-e1a97e052456","Type":"ContainerDied","Data":"cb24d3bb61c75888e6c98774411bb4342e38f631fa82c19bcb4513e26e305386"}
Apr 17 16:41:44.454087 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.453982    2579 scope.go:117] "RemoveContainer" containerID="5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"
Apr 17 16:41:44.463278 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.463259    2579 scope.go:117] "RemoveContainer" containerID="5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"
Apr 17 16:41:44.463535 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:44.463517    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2\": container with ID starting with 5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2 not found: ID does not exist" containerID="5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"
Apr 17 16:41:44.463579 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.463543    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2"} err="failed to get container status \"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2\": rpc error: code = NotFound desc = could not find container \"5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2\": container with ID starting with 5c8f508084f63b924206267b64e58135ee10e8ac476cf056bfa3cdce37fbafa2 not found: ID does not exist"
Apr 17 16:41:44.475114 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.475089    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:44.476697 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:44.476677    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-fb5qt"]
Apr 17 16:41:45.389346 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:45.389316    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406c0305-7dc6-422b-9cad-e1a97e052456" path="/var/lib/kubelet/pods/406c0305-7dc6-422b-9cad-e1a97e052456/volumes"
Apr 17 16:41:54.421648 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.421616    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"]
Apr 17 16:41:54.422073 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.422055    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406c0305-7dc6-422b-9cad-e1a97e052456" containerName="seaweedfs-tls-custom"
Apr 17 16:41:54.422117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.422077    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="406c0305-7dc6-422b-9cad-e1a97e052456" containerName="seaweedfs-tls-custom"
Apr 17 16:41:54.422172 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.422162    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="406c0305-7dc6-422b-9cad-e1a97e052456" containerName="seaweedfs-tls-custom"
Apr 17 16:41:54.425682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.425666    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.428014 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.427987    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Apr 17 16:41:54.428289 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.428267    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Apr 17 16:41:54.428392 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.428268    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mdnc6\""
Apr 17 16:41:54.434398 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.434376    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"]
Apr 17 16:41:54.543213 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.543169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.543396 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.543234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4f192-2403-4ca3-8524-4b565f3a99b7-data\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.543396 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.543272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27v9k\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-kube-api-access-27v9k\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.644330 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.644291    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.644540 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.644341    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4f192-2403-4ca3-8524-4b565f3a99b7-data\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.644540 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.644377    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27v9k\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-kube-api-access-27v9k\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.644540 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:54.644441    2579 projected.go:264] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Apr 17 16:41:54.644540 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:54.644461    2579 projected.go:194] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-7fd5766db9-j6slw: secret "seaweedfs-tls-serving" not found
Apr 17 16:41:54.644540 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:41:54.644517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving podName:07e4f192-2403-4ca3-8524-4b565f3a99b7 nodeName:}" failed. No retries permitted until 2026-04-17 16:41:55.144498258 +0000 UTC m=+642.385156680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-7fd5766db9-j6slw" (UID: "07e4f192-2403-4ca3-8524-4b565f3a99b7") : secret "seaweedfs-tls-serving" not found
Apr 17 16:41:54.644885 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.644861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4f192-2403-4ca3-8524-4b565f3a99b7-data\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:54.655042 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:54.655013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27v9k\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-kube-api-access-27v9k\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:55.148235 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:55.148201    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:55.150534 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:55.150511    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07e4f192-2403-4ca3-8524-4b565f3a99b7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-j6slw\" (UID: \"07e4f192-2403-4ca3-8524-4b565f3a99b7\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:55.337244 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:55.337216    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"
Apr 17 16:41:55.460774 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:55.460738    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-j6slw"]
Apr 17 16:41:55.462916 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:41:55.462889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e4f192_2403_4ca3_8524_4b565f3a99b7.slice/crio-c9904f9a0c75ed11494e4f2b5c69d54f87f16a6cbe56623698580c004f5580fe WatchSource:0}: Error finding container c9904f9a0c75ed11494e4f2b5c69d54f87f16a6cbe56623698580c004f5580fe: Status 404 returned error can't find the container with id c9904f9a0c75ed11494e4f2b5c69d54f87f16a6cbe56623698580c004f5580fe
Apr 17 16:41:55.492440 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:55.492410    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw" event={"ID":"07e4f192-2403-4ca3-8524-4b565f3a99b7","Type":"ContainerStarted","Data":"c9904f9a0c75ed11494e4f2b5c69d54f87f16a6cbe56623698580c004f5580fe"}
Apr 17 16:41:56.497036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:56.497001    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw" event={"ID":"07e4f192-2403-4ca3-8524-4b565f3a99b7","Type":"ContainerStarted","Data":"3aff750370ed009f0942b43ac868bc76b10e608a670fc1785cb18fc3c707b1b3"}
Apr 17 16:41:56.511410 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:41:56.511364    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-j6slw" podStartSLOduration=2.2679016020000002 podStartE2EDuration="2.511352063s" podCreationTimestamp="2026-04-17 16:41:54 +0000 UTC" firstStartedPulling="2026-04-17 16:41:55.464122596 +0000 UTC m=+642.704781009" lastFinishedPulling="2026-04-17 16:41:55.707573057 +0000 UTC m=+642.948231470" observedRunningTime="2026-04-17 16:41:56.510269984 +0000 UTC m=+643.750928420" watchObservedRunningTime="2026-04-17 16:41:56.511352063 +0000 UTC m=+643.752010499"
Apr 17 16:42:13.953040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.953002    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:42:13.957017 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.956998    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:13.959103 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.959084    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Apr 17 16:42:13.959339 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.959316    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Apr 17 16:42:13.959339 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.959334    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 16:42:13.959513 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.959410    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 16:42:13.959662 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.959641    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 16:42:13.967883 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:13.967858    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:42:14.018879 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.018847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.019036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.018904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.019036 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.019000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxc7t\" (UniqueName: \"kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.019120 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.019042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.120319 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.120273    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.120510 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.120354    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.120510 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.120395    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxc7t\" (UniqueName: \"kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.120510 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.120423    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.120659 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:42:14.120606    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Apr 17 16:42:14.120715 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:42:14.120670    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls podName:3b1d2ca1-5872-4d90-869e-671b71f40731 nodeName:}" failed. No retries permitted until 2026-04-17 16:42:14.620651433 +0000 UTC m=+661.861309862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls") pod "isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" (UID: "3b1d2ca1-5872-4d90-869e-671b71f40731") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Apr 17 16:42:14.120815 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.120787    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.121066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.121049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.128646 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.128624    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxc7t\" (UniqueName: \"kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.625820 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.625747    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.628127 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.628107    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.868945 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.868907    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:14.999805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:14.999731    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:42:15.002567 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:42:15.002536    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1d2ca1_5872_4d90_869e_671b71f40731.slice/crio-dcf13fe7ed0d5227bdd7b40052cb25c9250a22d52c1eff1cceb04343c1a30ae9 WatchSource:0}: Error finding container dcf13fe7ed0d5227bdd7b40052cb25c9250a22d52c1eff1cceb04343c1a30ae9: Status 404 returned error can't find the container with id dcf13fe7ed0d5227bdd7b40052cb25c9250a22d52c1eff1cceb04343c1a30ae9
Apr 17 16:42:15.562985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:15.562949    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerStarted","Data":"dcf13fe7ed0d5227bdd7b40052cb25c9250a22d52c1eff1cceb04343c1a30ae9"}
Apr 17 16:42:18.575159 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:18.575073    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerStarted","Data":"29bdaa29e3f52601e1297208bd3bcdc95c5a152e47c1c9437deb8f15dfec1c5d"}
Apr 17 16:42:22.589066 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:22.589030    2579 generic.go:358] "Generic (PLEG): container finished" podID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerID="29bdaa29e3f52601e1297208bd3bcdc95c5a152e47c1c9437deb8f15dfec1c5d" exitCode=0
Apr 17 16:42:22.589471 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:22.589104    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerDied","Data":"29bdaa29e3f52601e1297208bd3bcdc95c5a152e47c1c9437deb8f15dfec1c5d"}
Apr 17 16:42:36.640962 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:36.640927    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerStarted","Data":"4104d5808c071a700ae14395c41baf0e9a1c7e61f24ae22a3b6eb6d45c0ec1cb"}
Apr 17 16:42:38.650627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:38.650593    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerStarted","Data":"4ea4eeed65bb3526968763a466de7e52155c8e8bd616078c51f7bdc9d10c4c9b"}
Apr 17 16:42:41.662746 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:41.662703    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerStarted","Data":"58a0406cf37deaa02f998e4c35fa2005c54134dd961312da591db39d958054a9"}
Apr 17 16:42:41.663179 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:41.662960    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:41.663179 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:41.663067    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:41.664167 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:41.664139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:42:41.683875 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:41.683829    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podStartSLOduration=2.34932536 podStartE2EDuration="28.683816181s" podCreationTimestamp="2026-04-17 16:42:13 +0000 UTC" firstStartedPulling="2026-04-17 16:42:15.005124098 +0000 UTC m=+662.245782528" lastFinishedPulling="2026-04-17 16:42:41.339614935 +0000 UTC m=+688.580273349" observedRunningTime="2026-04-17 16:42:41.681598365 +0000 UTC m=+688.922256802" watchObservedRunningTime="2026-04-17 16:42:41.683816181 +0000 UTC m=+688.924474620"
Apr 17 16:42:42.666618 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:42.666569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:42:42.667059 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:42.666591    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:42.667598 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:42.667573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:42:43.670431 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:43.670391    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:42:43.670961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:43.670804    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:42:43.674028 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:43.674009    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:42:44.674553 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:44.674508    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:42:44.675039 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:44.674819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:42:54.675431 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:54.675385    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:42:54.675948 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:42:54.675836    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:43:04.675259 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:04.675205    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:43:04.675655 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:04.675627    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:43:14.675472 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:14.675417    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:43:14.675948 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:14.675900    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:43:24.674527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:24.674476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:43:24.675095 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:24.674901    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:43:34.675259 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:34.675216    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:43:34.675691 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:34.675607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:43:44.674972 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:44.674937    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:43:44.675396 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:44.675194    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:43:59.004845 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.004808    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:43:59.005661 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.005600    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" containerID="cri-o://58a0406cf37deaa02f998e4c35fa2005c54134dd961312da591db39d958054a9" gracePeriod=30
Apr 17 16:43:59.005813 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.005797    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" containerID="cri-o://4ea4eeed65bb3526968763a466de7e52155c8e8bd616078c51f7bdc9d10c4c9b" gracePeriod=30
Apr 17 16:43:59.005955 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.005909    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" containerID="cri-o://4104d5808c071a700ae14395c41baf0e9a1c7e61f24ae22a3b6eb6d45c0ec1cb" gracePeriod=30
Apr 17 16:43:59.113263 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.113220    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:43:59.117681 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.117658    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.119845 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.119822    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Apr 17 16:43:59.119845 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.119836    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Apr 17 16:43:59.128222 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.128199    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:43:59.172390 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.172340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.172604 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.172410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.172604 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.172441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfr4\" (UniqueName: \"kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.172604 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.172558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.273977 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.273873    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfr4\" (UniqueName: \"kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.273977 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.273965    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.274201 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.274022    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.274201 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.274067    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.274201 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:43:59.274171    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Apr 17 16:43:59.274317 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:43:59.274233    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls podName:f39e851a-b16f-4c87-9593-2cee656ad54c nodeName:}" failed. No retries permitted until 2026-04-17 16:43:59.774211441 +0000 UTC m=+767.014869876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Apr 17 16:43:59.274403 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.274383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.274700 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.274680    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.282106 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.282076    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfr4\" (UniqueName: \"kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.780340 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.780296    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:43:59.780540 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:43:59.780446    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Apr 17 16:43:59.780540 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:43:59.780513    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls podName:f39e851a-b16f-4c87-9593-2cee656ad54c nodeName:}" failed. No retries permitted until 2026-04-17 16:44:00.780497473 +0000 UTC m=+768.021155890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Apr 17 16:43:59.929646 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.929615    2579 generic.go:358] "Generic (PLEG): container finished" podID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerID="4ea4eeed65bb3526968763a466de7e52155c8e8bd616078c51f7bdc9d10c4c9b" exitCode=2
Apr 17 16:43:59.929814 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:43:59.929690    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerDied","Data":"4ea4eeed65bb3526968763a466de7e52155c8e8bd616078c51f7bdc9d10c4c9b"}
Apr 17 16:44:00.790059 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:00.790025    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:00.792547 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:00.792522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:00.929937 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:00.929904    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:01.054392 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:01.054368    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:44:01.056746 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:44:01.056722    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39e851a_b16f_4c87_9593_2cee656ad54c.slice/crio-77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6 WatchSource:0}: Error finding container 77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6: Status 404 returned error can't find the container with id 77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6
Apr 17 16:44:01.058668 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:01.058647    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 16:44:01.937150 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:01.937119    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerStarted","Data":"f44a01dde1d85069ba0c10b350ea6fa0c787f20ce7d079ab93ea0df41e015f1b"}
Apr 17 16:44:01.937150 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:01.937151    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerStarted","Data":"77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6"}
Apr 17 16:44:03.670730 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:03.670634    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:03.945805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:03.945698    2579 generic.go:358] "Generic (PLEG): container finished" podID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerID="4104d5808c071a700ae14395c41baf0e9a1c7e61f24ae22a3b6eb6d45c0ec1cb" exitCode=0
Apr 17 16:44:03.945805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:03.945785    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerDied","Data":"4104d5808c071a700ae14395c41baf0e9a1c7e61f24ae22a3b6eb6d45c0ec1cb"}
Apr 17 16:44:04.674863 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:04.674814    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:44:04.675253 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:04.675110    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:04.951166 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:04.951079    2579 generic.go:358] "Generic (PLEG): container finished" podID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerID="f44a01dde1d85069ba0c10b350ea6fa0c787f20ce7d079ab93ea0df41e015f1b" exitCode=0
Apr 17 16:44:04.951332 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:04.951156    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerDied","Data":"f44a01dde1d85069ba0c10b350ea6fa0c787f20ce7d079ab93ea0df41e015f1b"}
Apr 17 16:44:05.957775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.957727    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerStarted","Data":"f6c7a46e3bc64290f6be94d121894462b59813adb423518c87cf02fc6527d208"}
Apr 17 16:44:05.957775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.957780    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerStarted","Data":"e0b4be07eb7410d9b73ea022261ea9fb0611bbb42261a816521a7acdfd83884e"}
Apr 17 16:44:05.958249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.957792    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerStarted","Data":"65b8102d15a578749f0feaccbdb1e0a2070eb7f821f4d1f9f212f79c5626ed75"}
Apr 17 16:44:05.958249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.958097    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:05.958249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.958127    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:05.958249 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.958139    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:05.959720 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.959691    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:05.960360 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.960339    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:05.978052 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:05.978015    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podStartSLOduration=6.978002924 podStartE2EDuration="6.978002924s" podCreationTimestamp="2026-04-17 16:43:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:44:05.975936194 +0000 UTC m=+773.216594629" watchObservedRunningTime="2026-04-17 16:44:05.978002924 +0000 UTC m=+773.218661360"
Apr 17 16:44:06.961054 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:06.961011    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:06.961441 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:06.961398    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:08.670952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:08.670912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:11.964621 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:11.964587    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:44:11.965246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:11.965218    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:11.965645 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:11.965617    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:13.671255 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:13.671207    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:13.671707 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:13.671361    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:44:14.675097 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:14.675047    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:44:14.675527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:14.675350    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:18.670917 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:18.670875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:21.965791 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:21.965740    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:21.966215 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:21.966171    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:23.670952 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:23.670906    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:24.675204 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:24.675151    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 17 16:44:24.675667 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:24.675310    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:44:24.675667 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:24.675448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:24.675667 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:24.675555    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:44:28.670985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:28.670889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Apr 17 16:44:29.036235 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.036203    2579 generic.go:358] "Generic (PLEG): container finished" podID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerID="58a0406cf37deaa02f998e4c35fa2005c54134dd961312da591db39d958054a9" exitCode=0
Apr 17 16:44:29.036366 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.036261    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerDied","Data":"58a0406cf37deaa02f998e4c35fa2005c54134dd961312da591db39d958054a9"}
Apr 17 16:44:29.168091 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.168069    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:44:29.354966 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.354874    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location\") pod \"3b1d2ca1-5872-4d90-869e-671b71f40731\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") "
Apr 17 16:44:29.354966 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.354917    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxc7t\" (UniqueName: \"kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t\") pod \"3b1d2ca1-5872-4d90-869e-671b71f40731\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") "
Apr 17 16:44:29.355196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.355014    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"3b1d2ca1-5872-4d90-869e-671b71f40731\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") "
Apr 17 16:44:29.355196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.355070    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") pod \"3b1d2ca1-5872-4d90-869e-671b71f40731\" (UID: \"3b1d2ca1-5872-4d90-869e-671b71f40731\") "
Apr 17 16:44:29.355304 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.355188    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b1d2ca1-5872-4d90-869e-671b71f40731" (UID: "3b1d2ca1-5872-4d90-869e-671b71f40731"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:44:29.355456 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.355424    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1d2ca1-5872-4d90-869e-671b71f40731-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:44:29.355456 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.355427    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "3b1d2ca1-5872-4d90-869e-671b71f40731" (UID: "3b1d2ca1-5872-4d90-869e-671b71f40731"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:44:29.357182 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.357161    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b1d2ca1-5872-4d90-869e-671b71f40731" (UID: "3b1d2ca1-5872-4d90-869e-671b71f40731"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:44:29.357282 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.357266    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t" (OuterVolumeSpecName: "kube-api-access-pxc7t") pod "3b1d2ca1-5872-4d90-869e-671b71f40731" (UID: "3b1d2ca1-5872-4d90-869e-671b71f40731"). InnerVolumeSpecName "kube-api-access-pxc7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:44:29.455916 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.455878    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1d2ca1-5872-4d90-869e-671b71f40731-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:44:29.455916 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.455908    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1d2ca1-5872-4d90-869e-671b71f40731-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:44:29.455916 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:29.455919    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxc7t\" (UniqueName: \"kubernetes.io/projected/3b1d2ca1-5872-4d90-869e-671b71f40731-kube-api-access-pxc7t\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:44:30.041545 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.041510    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz" event={"ID":"3b1d2ca1-5872-4d90-869e-671b71f40731","Type":"ContainerDied","Data":"dcf13fe7ed0d5227bdd7b40052cb25c9250a22d52c1eff1cceb04343c1a30ae9"}
Apr 17 16:44:30.042147 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.041562    2579 scope.go:117] "RemoveContainer" containerID="58a0406cf37deaa02f998e4c35fa2005c54134dd961312da591db39d958054a9"
Apr 17 16:44:30.042147 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.041521    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"
Apr 17 16:44:30.049709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.049693    2579 scope.go:117] "RemoveContainer" containerID="4ea4eeed65bb3526968763a466de7e52155c8e8bd616078c51f7bdc9d10c4c9b"
Apr 17 16:44:30.056622 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.056600    2579 scope.go:117] "RemoveContainer" containerID="4104d5808c071a700ae14395c41baf0e9a1c7e61f24ae22a3b6eb6d45c0ec1cb"
Apr 17 16:44:30.060525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.060503    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:44:30.064458 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.064435    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-vjpxz"]
Apr 17 16:44:30.065280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:30.065268    2579 scope.go:117] "RemoveContainer" containerID="29bdaa29e3f52601e1297208bd3bcdc95c5a152e47c1c9437deb8f15dfec1c5d"
Apr 17 16:44:31.389193 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:31.389154    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" path="/var/lib/kubelet/pods/3b1d2ca1-5872-4d90-869e-671b71f40731/volumes"
Apr 17 16:44:31.966014 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:31.965973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:31.966477 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:31.966454    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:41.965279 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:41.965237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:41.965707 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:41.965535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:44:51.965884 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:51.965843    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:44:51.966341 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:44:51.966316    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:01.965220 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:01.965174    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:45:01.965716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:01.965617    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:11.966493 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:11.966454    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:11.966963 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:11.966677    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:24.199816 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.199785    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:45:24.200292 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.200189    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" containerID="cri-o://65b8102d15a578749f0feaccbdb1e0a2070eb7f821f4d1f9f212f79c5626ed75" gracePeriod=30
Apr 17 16:45:24.200377 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.200352    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" containerID="cri-o://f6c7a46e3bc64290f6be94d121894462b59813adb423518c87cf02fc6527d208" gracePeriod=30
Apr 17 16:45:24.200604 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.200394    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" containerID="cri-o://e0b4be07eb7410d9b73ea022261ea9fb0611bbb42261a816521a7acdfd83884e" gracePeriod=30
Apr 17 16:45:24.265698 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.265663    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:45:24.266184 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266168    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="storage-initializer"
Apr 17 16:45:24.266234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="storage-initializer"
Apr 17 16:45:24.266234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266202    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container"
Apr 17 16:45:24.266234 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266211    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266234    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266243    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266263    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266332    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="agent"
Apr 17 16:45:24.266354 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266348    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kserve-container"
Apr 17 16:45:24.266526 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.266361    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b1d2ca1-5872-4d90-869e-671b71f40731" containerName="kube-rbac-proxy"
Apr 17 16:45:24.271115 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.271096    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.273457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.273438    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Apr 17 16:45:24.273721 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.273704    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Apr 17 16:45:24.281683 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.281658    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:45:24.409058 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.409025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.409244 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.409104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfwj\" (UniqueName: \"kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.409244 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.409142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.510534 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.510447    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.510703 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.510541    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-plfwj\" (UniqueName: \"kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.510703 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.510578    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.511236 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.511211    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.512986 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.512967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.520548 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.520526    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfwj\" (UniqueName: \"kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj\") pod \"message-dumper-predictor-c7d86bcbd-hhjq9\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.583082 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.583053    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:24.704465 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:24.704414    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:45:24.706808 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:45:24.706775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044ce7fd_95e3_4b7a_bda1_bc3a7a80d9f0.slice/crio-b3a9346340d92d62833c2dd6a596d954c03a2978217c0f13a9311d567ff0e4ac WatchSource:0}: Error finding container b3a9346340d92d62833c2dd6a596d954c03a2978217c0f13a9311d567ff0e4ac: Status 404 returned error can't find the container with id b3a9346340d92d62833c2dd6a596d954c03a2978217c0f13a9311d567ff0e4ac
Apr 17 16:45:25.231546 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:25.231516    2579 generic.go:358] "Generic (PLEG): container finished" podID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerID="e0b4be07eb7410d9b73ea022261ea9fb0611bbb42261a816521a7acdfd83884e" exitCode=2
Apr 17 16:45:25.231948 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:25.231590    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerDied","Data":"e0b4be07eb7410d9b73ea022261ea9fb0611bbb42261a816521a7acdfd83884e"}
Apr 17 16:45:25.232643 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:25.232624    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerStarted","Data":"b3a9346340d92d62833c2dd6a596d954c03a2978217c0f13a9311d567ff0e4ac"}
Apr 17 16:45:26.238292 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:26.238254    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerStarted","Data":"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"}
Apr 17 16:45:26.238292 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:26.238295    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerStarted","Data":"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"}
Apr 17 16:45:26.238747 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:26.238391    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:26.259014 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:26.258973    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" podStartSLOduration=1.173811128 podStartE2EDuration="2.258954626s" podCreationTimestamp="2026-04-17 16:45:24 +0000 UTC" firstStartedPulling="2026-04-17 16:45:24.708674071 +0000 UTC m=+851.949332485" lastFinishedPulling="2026-04-17 16:45:25.793817566 +0000 UTC m=+853.034475983" observedRunningTime="2026-04-17 16:45:26.257635967 +0000 UTC m=+853.498294406" watchObservedRunningTime="2026-04-17 16:45:26.258954626 +0000 UTC m=+853.499613062"
Apr 17 16:45:26.961930 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:26.961887    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:27.241850 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:27.241748    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:27.243244 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:27.243220    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:29.250645 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:29.250609    2579 generic.go:358] "Generic (PLEG): container finished" podID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerID="65b8102d15a578749f0feaccbdb1e0a2070eb7f821f4d1f9f212f79c5626ed75" exitCode=0
Apr 17 16:45:29.251012 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:29.250677    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerDied","Data":"65b8102d15a578749f0feaccbdb1e0a2070eb7f821f4d1f9f212f79c5626ed75"}
Apr 17 16:45:31.961982 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:31.961936    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:31.965251 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:31.965220    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:45:31.965647 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:31.965624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:34.254924 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:34.254893    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:45:36.961549 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:36.961503    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:36.961960 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:36.961652    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:41.961897 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:41.961851    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:41.965146 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:41.965113    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:45:41.965469 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:41.965446    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:44.322186 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.322156    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:45:44.326107 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.326086    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.328325 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.328301    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Apr 17 16:45:44.328459 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.328304    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Apr 17 16:45:44.335793 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.335755    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:45:44.372617 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.372589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.372790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.372636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.372790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.372678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99pph\" (UniqueName: \"kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.372790 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.372723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.473567 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.473536    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99pph\" (UniqueName: \"kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.473775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.473698    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.473889 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.473868    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.474000 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.473983    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.474113 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:45:44.474098    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Apr 17 16:45:44.474177 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:45:44.474167    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls podName:eed7ef02-490b-4d87-9417-4127ca78264f nodeName:}" failed. No retries permitted until 2026-04-17 16:45:44.974146964 +0000 UTC m=+872.214805393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls") pod "isvc-logger-predictor-64d54fcc88-5j2l7" (UID: "eed7ef02-490b-4d87-9417-4127ca78264f") : secret "isvc-logger-predictor-serving-cert" not found
Apr 17 16:45:44.474385 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.474365    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.474671 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.474648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.482681 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.482659    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99pph\" (UniqueName: \"kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.977344 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.977309    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:44.979726 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:44.979704    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") pod \"isvc-logger-predictor-64d54fcc88-5j2l7\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:45.238332 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:45.238250    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:45.363844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:45.363821    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:45:45.366415 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:45:45.366390    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed7ef02_490b_4d87_9417_4127ca78264f.slice/crio-c0eee034ff40fa4ac419eb3274249563520eb1c2b62dfe2cef277300e136b41c WatchSource:0}: Error finding container c0eee034ff40fa4ac419eb3274249563520eb1c2b62dfe2cef277300e136b41c: Status 404 returned error can't find the container with id c0eee034ff40fa4ac419eb3274249563520eb1c2b62dfe2cef277300e136b41c
Apr 17 16:45:46.308961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:46.308922    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerStarted","Data":"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"}
Apr 17 16:45:46.308961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:46.308963    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerStarted","Data":"c0eee034ff40fa4ac419eb3274249563520eb1c2b62dfe2cef277300e136b41c"}
Apr 17 16:45:46.961958 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:46.961909    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:49.319719 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:49.319688    2579 generic.go:358] "Generic (PLEG): container finished" podID="eed7ef02-490b-4d87-9417-4127ca78264f" containerID="b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8" exitCode=0
Apr 17 16:45:49.320099 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:49.319770    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerDied","Data":"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"}
Apr 17 16:45:50.325358 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:50.325321    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerStarted","Data":"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"}
Apr 17 16:45:50.325732 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:50.325367    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerStarted","Data":"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"}
Apr 17 16:45:50.325732 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:50.325378    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerStarted","Data":"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"}
Apr 17 16:45:50.325732 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:50.325595    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:50.344171 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:50.344122    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podStartSLOduration=6.344108757 podStartE2EDuration="6.344108757s" podCreationTimestamp="2026-04-17 16:45:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:45:50.343966393 +0000 UTC m=+877.584624831" watchObservedRunningTime="2026-04-17 16:45:50.344108757 +0000 UTC m=+877.584767192"
Apr 17 16:45:51.329357 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.329326    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:51.329357 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.329366    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:51.330816 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.330776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:45:51.331426 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.331401    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:51.961613 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.961564    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Apr 17 16:45:51.966050 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.966028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:5000: connect: connection refused"
Apr 17 16:45:51.966156 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.966141    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:51.966400 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.966374    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:51.966486 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:51.966475    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:52.333027 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:52.332932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:45:52.333379 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:52.333320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:45:54.341714 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.341585    2579 generic.go:358] "Generic (PLEG): container finished" podID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerID="f6c7a46e3bc64290f6be94d121894462b59813adb423518c87cf02fc6527d208" exitCode=0
Apr 17 16:45:54.341714 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.341650    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerDied","Data":"f6c7a46e3bc64290f6be94d121894462b59813adb423518c87cf02fc6527d208"}
Apr 17 16:45:54.341714 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.341683    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb" event={"ID":"f39e851a-b16f-4c87-9593-2cee656ad54c","Type":"ContainerDied","Data":"77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6"}
Apr 17 16:45:54.341714 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.341693    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77cce67b6599528b2b8bd963643157681627f7d38c98e948705c8dbd56ff3eb6"
Apr 17 16:45:54.355211 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.355193    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:54.446321 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446290    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"f39e851a-b16f-4c87-9593-2cee656ad54c\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") "
Apr 17 16:45:54.446457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446391    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pfr4\" (UniqueName: \"kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4\") pod \"f39e851a-b16f-4c87-9593-2cee656ad54c\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") "
Apr 17 16:45:54.446457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446411    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location\") pod \"f39e851a-b16f-4c87-9593-2cee656ad54c\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") "
Apr 17 16:45:54.446457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446430    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") pod \"f39e851a-b16f-4c87-9593-2cee656ad54c\" (UID: \"f39e851a-b16f-4c87-9593-2cee656ad54c\") "
Apr 17 16:45:54.446735 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446687    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "f39e851a-b16f-4c87-9593-2cee656ad54c" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:45:54.446735 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.446718    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f39e851a-b16f-4c87-9593-2cee656ad54c" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:45:54.448575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.448548    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f39e851a-b16f-4c87-9593-2cee656ad54c" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:45:54.448575 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.448567    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4" (OuterVolumeSpecName: "kube-api-access-7pfr4") pod "f39e851a-b16f-4c87-9593-2cee656ad54c" (UID: "f39e851a-b16f-4c87-9593-2cee656ad54c"). InnerVolumeSpecName "kube-api-access-7pfr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:45:54.547056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.546980    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfr4\" (UniqueName: \"kubernetes.io/projected/f39e851a-b16f-4c87-9593-2cee656ad54c-kube-api-access-7pfr4\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:45:54.547056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.547006    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f39e851a-b16f-4c87-9593-2cee656ad54c-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:45:54.547056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.547020    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f39e851a-b16f-4c87-9593-2cee656ad54c-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:45:54.547056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:54.547033    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f39e851a-b16f-4c87-9593-2cee656ad54c-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:45:55.345008 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:55.344979    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"
Apr 17 16:45:55.367473 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:55.367447    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:45:55.371099 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:55.371076    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-cxrqb"]
Apr 17 16:45:55.388875 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:55.388839    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" path="/var/lib/kubelet/pods/f39e851a-b16f-4c87-9593-2cee656ad54c/volumes"
Apr 17 16:45:57.336274 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:57.336244    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:45:57.336938 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:57.336909    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:45:57.337227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:45:57.337208    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:07.337265 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:07.337222    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:46:07.337827 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:07.337709    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:13.336869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:13.336837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:46:13.338596 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:13.338572    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:46:13.340129 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:13.340112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:46:13.341668 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:13.341650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:46:17.336905 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:17.336868    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:46:17.337913 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:17.337879    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:27.336889 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:27.336838    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:46:27.340838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:27.337207    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:37.336868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:37.336819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:46:37.337280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:37.337250    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:47.336895 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:47.336851    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:46:47.337416 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:47.337371    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:46:57.337635 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:57.337600    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:46:57.338046 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:46:57.337667    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:09.353277 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.353238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-hhjq9_044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0/kserve-container/0.log"
Apr 17 16:47:09.564276 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.564239    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:47:09.564782 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.564695    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" containerID="cri-o://ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941" gracePeriod=30
Apr 17 16:47:09.564907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.564748    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" containerID="cri-o://2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2" gracePeriod=30
Apr 17 16:47:09.564907 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.564735    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" containerID="cri-o://eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460" gracePeriod=30
Apr 17 16:47:09.654462 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.654433    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:47:09.660886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.660858    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container"
Apr 17 16:47:09.660886 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.660881    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.660931    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="storage-initializer"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.660976    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="storage-initializer"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661005    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661045    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent"
Apr 17 16:47:09.661138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661088    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent"
Apr 17 16:47:09.661425 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661394    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="agent"
Apr 17 16:47:09.661478 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661417    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kube-rbac-proxy"
Apr 17 16:47:09.661478 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.661457    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f39e851a-b16f-4c87-9593-2cee656ad54c" containerName="kserve-container"
Apr 17 16:47:09.665349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.665330    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.670442 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.670259    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Apr 17 16:47:09.670629 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.670606    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Apr 17 16:47:09.672248 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.672215    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:47:09.750246 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.750217    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:47:09.750528 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.750503    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kserve-container" containerID="cri-o://b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695" gracePeriod=30
Apr 17 16:47:09.750599 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.750533    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kube-rbac-proxy" containerID="cri-o://7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e" gracePeriod=30
Apr 17 16:47:09.801632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.801597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.801753 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.801641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqgx\" (UniqueName: \"kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.801753 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.801741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.801893 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.801828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.902995 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.902966    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.903137 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.903027    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.903137 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.903056    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqgx\" (UniqueName: \"kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.903137 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.903122    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.903395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.903373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.903779 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.903748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.906149 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.906091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.913646 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.913620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqgx\" (UniqueName: \"kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx\") pod \"isvc-lightgbm-predictor-bdf964bd-c5x67\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.981329 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.981295    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:09.997638 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:09.997619    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:47:10.104876 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.104850    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls\") pod \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") "
Apr 17 16:47:10.105020 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.104938    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plfwj\" (UniqueName: \"kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj\") pod \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") "
Apr 17 16:47:10.105020 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.104988    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config\") pod \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\" (UID: \"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0\") "
Apr 17 16:47:10.105342 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.105320    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" (UID: "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:47:10.107379 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.107348    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj" (OuterVolumeSpecName: "kube-api-access-plfwj") pod "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" (UID: "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0"). InnerVolumeSpecName "kube-api-access-plfwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:47:10.107579 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.107466    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" (UID: "044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:47:10.109044 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.109023    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:47:10.111028 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:47:10.111000    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9206f02a_84ee_49bb_b6bd_ea80e20b29c8.slice/crio-39c79ace4a1a92f02cc53609ee0bb87485189d6fc400b32b8062cde150afecc0 WatchSource:0}: Error finding container 39c79ace4a1a92f02cc53609ee0bb87485189d6fc400b32b8062cde150afecc0: Status 404 returned error can't find the container with id 39c79ace4a1a92f02cc53609ee0bb87485189d6fc400b32b8062cde150afecc0
Apr 17 16:47:10.206373 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.206342    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:10.206373 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.206370    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfwj\" (UniqueName: \"kubernetes.io/projected/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-kube-api-access-plfwj\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:10.206555 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.206381    2579 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:10.611208 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611175    2579 generic.go:358] "Generic (PLEG): container finished" podID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerID="7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e" exitCode=2
Apr 17 16:47:10.611208 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611204    2579 generic.go:358] "Generic (PLEG): container finished" podID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerID="b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695" exitCode=2
Apr 17 16:47:10.611709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611260    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerDied","Data":"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"}
Apr 17 16:47:10.611709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611300    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerDied","Data":"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"}
Apr 17 16:47:10.611709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611310    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9" event={"ID":"044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0","Type":"ContainerDied","Data":"b3a9346340d92d62833c2dd6a596d954c03a2978217c0f13a9311d567ff0e4ac"}
Apr 17 16:47:10.611709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611327    2579 scope.go:117] "RemoveContainer" containerID="7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"
Apr 17 16:47:10.611709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.611273    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"
Apr 17 16:47:10.612888 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.612861    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerStarted","Data":"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"}
Apr 17 16:47:10.612994 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.612897    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerStarted","Data":"39c79ace4a1a92f02cc53609ee0bb87485189d6fc400b32b8062cde150afecc0"}
Apr 17 16:47:10.615471 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.615393    2579 generic.go:358] "Generic (PLEG): container finished" podID="eed7ef02-490b-4d87-9417-4127ca78264f" containerID="2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2" exitCode=2
Apr 17 16:47:10.615471 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.615450    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerDied","Data":"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"}
Apr 17 16:47:10.622248 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.622225    2579 scope.go:117] "RemoveContainer" containerID="b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"
Apr 17 16:47:10.629514 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.629495    2579 scope.go:117] "RemoveContainer" containerID="7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"
Apr 17 16:47:10.629725 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:10.629708    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e\": container with ID starting with 7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e not found: ID does not exist" containerID="7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"
Apr 17 16:47:10.629814 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.629733    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"} err="failed to get container status \"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e\": rpc error: code = NotFound desc = could not find container \"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e\": container with ID starting with 7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e not found: ID does not exist"
Apr 17 16:47:10.629814 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.629749    2579 scope.go:117] "RemoveContainer" containerID="b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"
Apr 17 16:47:10.629939 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:10.629923    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695\": container with ID starting with b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695 not found: ID does not exist" containerID="b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"
Apr 17 16:47:10.630013 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.629942    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"} err="failed to get container status \"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695\": rpc error: code = NotFound desc = could not find container \"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695\": container with ID starting with b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695 not found: ID does not exist"
Apr 17 16:47:10.630013 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.629954    2579 scope.go:117] "RemoveContainer" containerID="7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"
Apr 17 16:47:10.630121 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.630109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e"} err="failed to get container status \"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e\": rpc error: code = NotFound desc = could not find container \"7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e\": container with ID starting with 7343b126af3ab573f4fe2fb64e69330e7ebdfb9ffb9da98e704d852c499eb89e not found: ID does not exist"
Apr 17 16:47:10.630217 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.630123    2579 scope.go:117] "RemoveContainer" containerID="b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"
Apr 17 16:47:10.630323 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.630286    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695"} err="failed to get container status \"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695\": rpc error: code = NotFound desc = could not find container \"b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695\": container with ID starting with b28bc678ea5ed924693fb8a81e3ae218849dd890b3d2c895e4703106ec3dd695 not found: ID does not exist"
Apr 17 16:47:10.652069 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.652041    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:47:10.657183 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:10.657158    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-hhjq9"]
Apr 17 16:47:11.389536 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:11.389496    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" path="/var/lib/kubelet/pods/044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0/volumes"
Apr 17 16:47:12.333516 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:12.333476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:14.632893 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:14.632860    2579 generic.go:358] "Generic (PLEG): container finished" podID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerID="af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45" exitCode=0
Apr 17 16:47:14.633235 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:14.632933    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerDied","Data":"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"}
Apr 17 16:47:14.635190 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:14.635167    2579 generic.go:358] "Generic (PLEG): container finished" podID="eed7ef02-490b-4d87-9417-4127ca78264f" containerID="ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941" exitCode=0
Apr 17 16:47:14.635283 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:14.635249    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerDied","Data":"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"}
Apr 17 16:47:17.333413 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:17.333362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:17.337716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:17.337678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:47:17.338243 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:17.338210    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:47:21.664023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:21.663992    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerStarted","Data":"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"}
Apr 17 16:47:21.664329 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:21.664038    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerStarted","Data":"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"}
Apr 17 16:47:21.664329 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:21.664234    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:21.681568 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:21.681519    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podStartSLOduration=5.787474534 podStartE2EDuration="12.681505163s" podCreationTimestamp="2026-04-17 16:47:09 +0000 UTC" firstStartedPulling="2026-04-17 16:47:14.634356895 +0000 UTC m=+961.875015312" lastFinishedPulling="2026-04-17 16:47:21.528387526 +0000 UTC m=+968.769045941" observedRunningTime="2026-04-17 16:47:21.679719767 +0000 UTC m=+968.920378214" watchObservedRunningTime="2026-04-17 16:47:21.681505163 +0000 UTC m=+968.922163598"
Apr 17 16:47:22.333520 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:22.333478    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:22.333691 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:22.333593    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:22.667344 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:22.667310    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:22.668638 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:22.668610    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:47:23.670341 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:23.670306    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:47:27.333175 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:27.333084    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:27.337482 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:27.337451    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:47:27.337735 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:27.337704    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:47:28.675152 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:28.675123    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:47:28.675826 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:28.675797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:47:32.333052 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:32.333013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:37.333735 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:37.333695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 17 16:47:37.337019 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:37.336988    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 17 16:47:37.337166 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:37.337090    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:37.337371 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:37.337347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 17 16:47:37.337473 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:37.337429    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:38.676657 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:38.676626    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:47:39.716317 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.716293    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:39.724347 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.724315    2579 generic.go:358] "Generic (PLEG): container finished" podID="eed7ef02-490b-4d87-9417-4127ca78264f" containerID="eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460" exitCode=0
Apr 17 16:47:39.724463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.724395    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerDied","Data":"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"}
Apr 17 16:47:39.724463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.724435    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7" event={"ID":"eed7ef02-490b-4d87-9417-4127ca78264f","Type":"ContainerDied","Data":"c0eee034ff40fa4ac419eb3274249563520eb1c2b62dfe2cef277300e136b41c"}
Apr 17 16:47:39.724463 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.724452    2579 scope.go:117] "RemoveContainer" containerID="eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"
Apr 17 16:47:39.724580 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.724405    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"
Apr 17 16:47:39.732345 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.732325    2579 scope.go:117] "RemoveContainer" containerID="2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"
Apr 17 16:47:39.749583 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.749544    2579 scope.go:117] "RemoveContainer" containerID="ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"
Apr 17 16:47:39.757418 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.757396    2579 scope.go:117] "RemoveContainer" containerID="b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"
Apr 17 16:47:39.764669 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.764650    2579 scope.go:117] "RemoveContainer" containerID="eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"
Apr 17 16:47:39.764957 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:39.764933    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460\": container with ID starting with eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460 not found: ID does not exist" containerID="eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"
Apr 17 16:47:39.765010 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.764970    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460"} err="failed to get container status \"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460\": rpc error: code = NotFound desc = could not find container \"eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460\": container with ID starting with eaa0f4dc32968ebb120e2ca07dbeb783f7bf3e6c296d776861f1bf42dd262460 not found: ID does not exist"
Apr 17 16:47:39.765010 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.764991    2579 scope.go:117] "RemoveContainer" containerID="2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"
Apr 17 16:47:39.765252 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:39.765229    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2\": container with ID starting with 2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2 not found: ID does not exist" containerID="2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"
Apr 17 16:47:39.765339 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.765255    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2"} err="failed to get container status \"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2\": rpc error: code = NotFound desc = could not find container \"2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2\": container with ID starting with 2a52cd7aab0d69bb13631d7d4198237e0f7a07fd37d12914c0e1bfc6be0a36e2 not found: ID does not exist"
Apr 17 16:47:39.765339 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.765272    2579 scope.go:117] "RemoveContainer" containerID="ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"
Apr 17 16:47:39.765537 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:39.765520    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941\": container with ID starting with ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941 not found: ID does not exist" containerID="ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"
Apr 17 16:47:39.765587 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.765541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941"} err="failed to get container status \"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941\": rpc error: code = NotFound desc = could not find container \"ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941\": container with ID starting with ede6ce7cc60b78d533503ba7b0291edbb4ad4a142f15bd4eeba38362e4405941 not found: ID does not exist"
Apr 17 16:47:39.765587 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.765554    2579 scope.go:117] "RemoveContainer" containerID="b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"
Apr 17 16:47:39.765824 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:47:39.765805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8\": container with ID starting with b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8 not found: ID does not exist" containerID="b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"
Apr 17 16:47:39.765913 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.765826    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8"} err="failed to get container status \"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8\": rpc error: code = NotFound desc = could not find container \"b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8\": container with ID starting with b009cdbf99227c6e66d84e7f75520a1d0be8780095b8bfc886d2fac813b868a8 not found: ID does not exist"
Apr 17 16:47:39.775115 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775095    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99pph\" (UniqueName: \"kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph\") pod \"eed7ef02-490b-4d87-9417-4127ca78264f\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") "
Apr 17 16:47:39.775231 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775201    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location\") pod \"eed7ef02-490b-4d87-9417-4127ca78264f\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") "
Apr 17 16:47:39.775292 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775230    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") pod \"eed7ef02-490b-4d87-9417-4127ca78264f\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") "
Apr 17 16:47:39.775292 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775250    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config\") pod \"eed7ef02-490b-4d87-9417-4127ca78264f\" (UID: \"eed7ef02-490b-4d87-9417-4127ca78264f\") "
Apr 17 16:47:39.775566 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775542    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eed7ef02-490b-4d87-9417-4127ca78264f" (UID: "eed7ef02-490b-4d87-9417-4127ca78264f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:47:39.775671 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.775645    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "eed7ef02-490b-4d87-9417-4127ca78264f" (UID: "eed7ef02-490b-4d87-9417-4127ca78264f"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:47:39.777168 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.777145    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eed7ef02-490b-4d87-9417-4127ca78264f" (UID: "eed7ef02-490b-4d87-9417-4127ca78264f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:47:39.777465 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.777432    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph" (OuterVolumeSpecName: "kube-api-access-99pph") pod "eed7ef02-490b-4d87-9417-4127ca78264f" (UID: "eed7ef02-490b-4d87-9417-4127ca78264f"). InnerVolumeSpecName "kube-api-access-99pph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:47:39.876585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.876548    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eed7ef02-490b-4d87-9417-4127ca78264f-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:39.876585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.876578    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eed7ef02-490b-4d87-9417-4127ca78264f-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:39.876585 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.876590    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eed7ef02-490b-4d87-9417-4127ca78264f-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:39.876829 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:39.876600    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99pph\" (UniqueName: \"kubernetes.io/projected/eed7ef02-490b-4d87-9417-4127ca78264f-kube-api-access-99pph\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:47:40.046147 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:40.046119    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:47:40.051413 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:40.051389    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-5j2l7"]
Apr 17 16:47:41.389170 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:41.389139    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" path="/var/lib/kubelet/pods/eed7ef02-490b-4d87-9417-4127ca78264f/volumes"
Apr 17 16:47:48.676116 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:48.676075    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:47:58.676303 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:47:58.676265    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:48:08.676495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:08.676461    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:48:18.675735 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:18.675693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:48:28.676658 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:28.676623    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 17 16:48:37.389119 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:37.389091    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:48:39.879869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:39.879837    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:48:39.880340 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:39.880130    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container" containerID="cri-o://828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf" gracePeriod=30
Apr 17 16:48:39.880340 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:39.880200    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kube-rbac-proxy" containerID="cri-o://432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9" gracePeriod=30
Apr 17 16:48:40.013301 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013267    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:48:40.013683 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013668    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kserve-container"
Apr 17 16:48:40.013683 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kserve-container"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013692    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="storage-initializer"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013698    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="storage-initializer"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013703    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013710    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013722    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013727    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013737    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013742    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kube-rbac-proxy"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013779    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kube-rbac-proxy"
Apr 17 16:48:40.013852 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013849    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="agent"
Apr 17 16:48:40.014370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013861    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kube-rbac-proxy"
Apr 17 16:48:40.014370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013870    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kserve-container"
Apr 17 16:48:40.014370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013878    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="eed7ef02-490b-4d87-9417-4127ca78264f" containerName="kube-rbac-proxy"
Apr 17 16:48:40.014370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.013888    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="044ce7fd-95e3-4b7a-bda1-bc3a7a80d9f0" containerName="kserve-container"
Apr 17 16:48:40.017055 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.017035    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.019121 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.019100    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Apr 17 16:48:40.019245 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.019124    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Apr 17 16:48:40.024958 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.024916    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:48:40.094349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.094318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnwg\" (UniqueName: \"kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.094349 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.094351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.094561 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.094386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.094561 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.094431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.195853 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.195749    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.196022 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.195854    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnwg\" (UniqueName: \"kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.196022 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.195878    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.196022 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.195906    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.196280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.196261    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.196402 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.196381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.198252 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.198232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.203904 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.203882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnwg\" (UniqueName: \"kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.328223 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.328183    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:40.449640 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.449614    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:48:40.452275 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:48:40.452248    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa73954_ebbb_4dde_8d14_8c964415beaf.slice/crio-76bfadbe9b477c02f3921075ffac00d59090ed1ee0de73fecffb6b35a30db92e WatchSource:0}: Error finding container 76bfadbe9b477c02f3921075ffac00d59090ed1ee0de73fecffb6b35a30db92e: Status 404 returned error can't find the container with id 76bfadbe9b477c02f3921075ffac00d59090ed1ee0de73fecffb6b35a30db92e
Apr 17 16:48:40.935652 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.935618    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerStarted","Data":"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"}
Apr 17 16:48:40.935652 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.935656    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerStarted","Data":"76bfadbe9b477c02f3921075ffac00d59090ed1ee0de73fecffb6b35a30db92e"}
Apr 17 16:48:40.937587 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.937562    2579 generic.go:358] "Generic (PLEG): container finished" podID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerID="432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9" exitCode=2
Apr 17 16:48:40.937704 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:40.937594    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerDied","Data":"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"}
Apr 17 16:48:43.671395 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:43.671356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Apr 17 16:48:44.621934 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.621908    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:48:44.736423 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.736389    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls\") pod \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") "
Apr 17 16:48:44.736877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.736468    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location\") pod \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") "
Apr 17 16:48:44.736877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.736531    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvqgx\" (UniqueName: \"kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx\") pod \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") "
Apr 17 16:48:44.736877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.736583    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\" (UID: \"9206f02a-84ee-49bb-b6bd-ea80e20b29c8\") "
Apr 17 16:48:44.736877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.736826    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9206f02a-84ee-49bb-b6bd-ea80e20b29c8" (UID: "9206f02a-84ee-49bb-b6bd-ea80e20b29c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:48:44.737055 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.737029    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "9206f02a-84ee-49bb-b6bd-ea80e20b29c8" (UID: "9206f02a-84ee-49bb-b6bd-ea80e20b29c8"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:48:44.738641 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.738611    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx" (OuterVolumeSpecName: "kube-api-access-cvqgx") pod "9206f02a-84ee-49bb-b6bd-ea80e20b29c8" (UID: "9206f02a-84ee-49bb-b6bd-ea80e20b29c8"). InnerVolumeSpecName "kube-api-access-cvqgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:48:44.738747 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.738659    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9206f02a-84ee-49bb-b6bd-ea80e20b29c8" (UID: "9206f02a-84ee-49bb-b6bd-ea80e20b29c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:48:44.837433 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.837403    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:48:44.837433 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.837427    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqgx\" (UniqueName: \"kubernetes.io/projected/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-kube-api-access-cvqgx\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:48:44.837433 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.837439    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:48:44.837650 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.837449    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9206f02a-84ee-49bb-b6bd-ea80e20b29c8-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:48:44.953069 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.953033    2579 generic.go:358] "Generic (PLEG): container finished" podID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerID="828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf" exitCode=0
Apr 17 16:48:44.953227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.953106    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"
Apr 17 16:48:44.953227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.953110    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerDied","Data":"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"}
Apr 17 16:48:44.953227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.953145    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67" event={"ID":"9206f02a-84ee-49bb-b6bd-ea80e20b29c8","Type":"ContainerDied","Data":"39c79ace4a1a92f02cc53609ee0bb87485189d6fc400b32b8062cde150afecc0"}
Apr 17 16:48:44.953227 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.953162    2579 scope.go:117] "RemoveContainer" containerID="432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"
Apr 17 16:48:44.954576 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.954545    2579 generic.go:358] "Generic (PLEG): container finished" podID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerID="f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b" exitCode=0
Apr 17 16:48:44.954665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.954574    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerDied","Data":"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"}
Apr 17 16:48:44.962581 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.962512    2579 scope.go:117] "RemoveContainer" containerID="828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"
Apr 17 16:48:44.969591 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.969572    2579 scope.go:117] "RemoveContainer" containerID="af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"
Apr 17 16:48:44.983479 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.983462    2579 scope.go:117] "RemoveContainer" containerID="432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"
Apr 17 16:48:44.983707 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:48:44.983692    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9\": container with ID starting with 432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9 not found: ID does not exist" containerID="432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"
Apr 17 16:48:44.983754 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.983715    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9"} err="failed to get container status \"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9\": rpc error: code = NotFound desc = could not find container \"432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9\": container with ID starting with 432af362c4088f19bfd9410ddd608be2f675d00e85a0e0907f3a5be779db64b9 not found: ID does not exist"
Apr 17 16:48:44.983754 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.983735    2579 scope.go:117] "RemoveContainer" containerID="828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"
Apr 17 16:48:44.983977 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:48:44.983958    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf\": container with ID starting with 828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf not found: ID does not exist" containerID="828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"
Apr 17 16:48:44.984026 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.983984    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf"} err="failed to get container status \"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf\": rpc error: code = NotFound desc = could not find container \"828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf\": container with ID starting with 828145d9b2cccf5be80f4b1f57804f4c16ee2bcc6a00b55fefb17d409182b2cf not found: ID does not exist"
Apr 17 16:48:44.984026 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.984001    2579 scope.go:117] "RemoveContainer" containerID="af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"
Apr 17 16:48:44.984207 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:48:44.984190    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45\": container with ID starting with af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45 not found: ID does not exist" containerID="af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"
Apr 17 16:48:44.984250 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.984211    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45"} err="failed to get container status \"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45\": rpc error: code = NotFound desc = could not find container \"af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45\": container with ID starting with af610d1851843581223de5a87205c86ae932793ce64fd62aac3108ab7af3cb45 not found: ID does not exist"
Apr 17 16:48:44.988004 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.987986    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:48:44.991536 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:44.991517    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-c5x67"]
Apr 17 16:48:45.389400 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.389369    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" path="/var/lib/kubelet/pods/9206f02a-84ee-49bb-b6bd-ea80e20b29c8/volumes"
Apr 17 16:48:45.961168 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.961138    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerStarted","Data":"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"}
Apr 17 16:48:45.961168 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.961171    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerStarted","Data":"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"}
Apr 17 16:48:45.961601 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.961444    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:45.961601 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.961552    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:45.962845 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.962821    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:48:45.980366 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:45.980321    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podStartSLOduration=6.980310379 podStartE2EDuration="6.980310379s" podCreationTimestamp="2026-04-17 16:48:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:48:45.977854694 +0000 UTC m=+1053.218513129" watchObservedRunningTime="2026-04-17 16:48:45.980310379 +0000 UTC m=+1053.220968815"
Apr 17 16:48:46.965139 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:46.965107    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:48:51.970154 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:51.970125    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:48:51.970781 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:48:51.970719    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:01.971284 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:01.971240    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:11.970996 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:11.970959    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:21.970859 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:21.970820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:31.971534 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:31.971470    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:41.971236 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:41.971194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:49:51.971014 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:49:51.970968    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:50:01.972023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:01.971994    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:50:10.370117 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.370073    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:50:10.370673 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.370623    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" containerID="cri-o://ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153" gracePeriod=30
Apr 17 16:50:10.370906 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.370857    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kube-rbac-proxy" containerID="cri-o://fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78" gracePeriod=30
Apr 17 16:50:10.477543 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477511    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:50:10.477902 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kube-rbac-proxy"
Apr 17 16:50:10.477954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477903    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kube-rbac-proxy"
Apr 17 16:50:10.477954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477925    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="storage-initializer"
Apr 17 16:50:10.477954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477930    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="storage-initializer"
Apr 17 16:50:10.477954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container"
Apr 17 16:50:10.477954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.477945    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container"
Apr 17 16:50:10.478096 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.478000    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kube-rbac-proxy"
Apr 17 16:50:10.478096 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.478007    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9206f02a-84ee-49bb-b6bd-ea80e20b29c8" containerName="kserve-container"
Apr 17 16:50:10.480882 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.480862    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.482859 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.482834    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 17 16:50:10.482956 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.482870    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Apr 17 16:50:10.489370 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.489351    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:50:10.582805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.582745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.582954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.582821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.582954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.582870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.582954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.582911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtck\" (UniqueName: \"kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.683409 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.683335    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.683409 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.683372    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.683643 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.683413    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.683643 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.683456    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtck\" (UniqueName: \"kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.683940 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.683910    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.684200 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.684181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.685729 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.685712    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.691496 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.691447    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtck\" (UniqueName: \"kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.792366 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.792328    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:50:10.915275 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.915246    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:50:10.918364 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:50:10.918336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a93556_738d_46ba_b2cc_984adcfb957d.slice/crio-6f27fbcb833d265e9654053c9e32b51631b748330d445461e584baacd4869909 WatchSource:0}: Error finding container 6f27fbcb833d265e9654053c9e32b51631b748330d445461e584baacd4869909: Status 404 returned error can't find the container with id 6f27fbcb833d265e9654053c9e32b51631b748330d445461e584baacd4869909
Apr 17 16:50:10.920238 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:10.920217    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 16:50:11.262092 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.262000    2579 generic.go:358] "Generic (PLEG): container finished" podID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerID="fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78" exitCode=2
Apr 17 16:50:11.262250 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.262084    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerDied","Data":"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"}
Apr 17 16:50:11.263432 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.263400    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerStarted","Data":"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"}
Apr 17 16:50:11.263584 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.263437    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerStarted","Data":"6f27fbcb833d265e9654053c9e32b51631b748330d445461e584baacd4869909"}
Apr 17 16:50:11.965961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.965914    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Apr 17 16:50:11.970808 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:11.970779    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Apr 17 16:50:13.388148 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:13.388117    2579 scope.go:117] "RemoveContainer" containerID="f6c7a46e3bc64290f6be94d121894462b59813adb423518c87cf02fc6527d208"
Apr 17 16:50:13.397701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:13.397677    2579 scope.go:117] "RemoveContainer" containerID="65b8102d15a578749f0feaccbdb1e0a2070eb7f821f4d1f9f212f79c5626ed75"
Apr 17 16:50:13.404838 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:13.404742    2579 scope.go:117] "RemoveContainer" containerID="f44a01dde1d85069ba0c10b350ea6fa0c787f20ce7d079ab93ea0df41e015f1b"
Apr 17 16:50:13.411827 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:13.411808    2579 scope.go:117] "RemoveContainer" containerID="e0b4be07eb7410d9b73ea022261ea9fb0611bbb42261a816521a7acdfd83884e"
Apr 17 16:50:15.019025 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.019005    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:50:15.124237 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124213    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mnwg\" (UniqueName: \"kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg\") pod \"9fa73954-ebbb-4dde-8d14-8c964415beaf\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") "
Apr 17 16:50:15.124381 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124299    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"9fa73954-ebbb-4dde-8d14-8c964415beaf\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") "
Apr 17 16:50:15.124381 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124344    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls\") pod \"9fa73954-ebbb-4dde-8d14-8c964415beaf\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") "
Apr 17 16:50:15.124457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124422    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location\") pod \"9fa73954-ebbb-4dde-8d14-8c964415beaf\" (UID: \"9fa73954-ebbb-4dde-8d14-8c964415beaf\") "
Apr 17 16:50:15.124800 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124744    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "9fa73954-ebbb-4dde-8d14-8c964415beaf" (UID: "9fa73954-ebbb-4dde-8d14-8c964415beaf"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:50:15.124921 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.124811    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fa73954-ebbb-4dde-8d14-8c964415beaf" (UID: "9fa73954-ebbb-4dde-8d14-8c964415beaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:50:15.126327 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.126298    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fa73954-ebbb-4dde-8d14-8c964415beaf" (UID: "9fa73954-ebbb-4dde-8d14-8c964415beaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:50:15.126435 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.126384    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg" (OuterVolumeSpecName: "kube-api-access-6mnwg") pod "9fa73954-ebbb-4dde-8d14-8c964415beaf" (UID: "9fa73954-ebbb-4dde-8d14-8c964415beaf"). InnerVolumeSpecName "kube-api-access-6mnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:50:15.225399 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.225372    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fa73954-ebbb-4dde-8d14-8c964415beaf-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:50:15.225399 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.225397    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fa73954-ebbb-4dde-8d14-8c964415beaf-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:50:15.225573 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.225407    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mnwg\" (UniqueName: \"kubernetes.io/projected/9fa73954-ebbb-4dde-8d14-8c964415beaf-kube-api-access-6mnwg\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:50:15.225573 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.225417    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fa73954-ebbb-4dde-8d14-8c964415beaf-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:50:15.277720 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.277688    2579 generic.go:358] "Generic (PLEG): container finished" podID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerID="ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153" exitCode=0
Apr 17 16:50:15.278023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.277790    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerDied","Data":"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"}
Apr 17 16:50:15.278023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.277834    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t" event={"ID":"9fa73954-ebbb-4dde-8d14-8c964415beaf","Type":"ContainerDied","Data":"76bfadbe9b477c02f3921075ffac00d59090ed1ee0de73fecffb6b35a30db92e"}
Apr 17 16:50:15.278023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.277802    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"
Apr 17 16:50:15.278023 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.277856    2579 scope.go:117] "RemoveContainer" containerID="fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"
Apr 17 16:50:15.279179 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.279157    2579 generic.go:358] "Generic (PLEG): container finished" podID="24a93556-738d-46ba-b2cc-984adcfb957d" containerID="f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148" exitCode=0
Apr 17 16:50:15.279280 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.279191    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerDied","Data":"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"}
Apr 17 16:50:15.285918 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.285811    2579 scope.go:117] "RemoveContainer" containerID="ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"
Apr 17 16:50:15.293388 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.293368    2579 scope.go:117] "RemoveContainer" containerID="f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"
Apr 17 16:50:15.300730 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.300713    2579 scope.go:117] "RemoveContainer" containerID="fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"
Apr 17 16:50:15.300999 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:50:15.300983    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78\": container with ID starting with fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78 not found: ID does not exist" containerID="fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"
Apr 17 16:50:15.301052 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.301006    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78"} err="failed to get container status \"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78\": rpc error: code = NotFound desc = could not find container \"fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78\": container with ID starting with fbf095316649b4ac34c34880eaf3553f4c514b5fa446c7ab37c598ebd8147f78 not found: ID does not exist"
Apr 17 16:50:15.301052 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.301022    2579 scope.go:117] "RemoveContainer" containerID="ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"
Apr 17 16:50:15.301242 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:50:15.301226    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153\": container with ID starting with ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153 not found: ID does not exist" containerID="ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"
Apr 17 16:50:15.301305 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.301252    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153"} err="failed to get container status \"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153\": rpc error: code = NotFound desc = could not find container \"ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153\": container with ID starting with ba23a21d1cdffaeefb1ddddeecc694a82af33a70327cdfa4ddf63a5eda264153 not found: ID does not exist"
Apr 17 16:50:15.301305 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.301275    2579 scope.go:117] "RemoveContainer" containerID="f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"
Apr 17 16:50:15.301503 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:50:15.301489    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b\": container with ID starting with f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b not found: ID does not exist" containerID="f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"
Apr 17 16:50:15.301541 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.301506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b"} err="failed to get container status \"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b\": rpc error: code = NotFound desc = could not find container \"f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b\": container with ID starting with f8a8a6757480839667b6c7a7aa872fc64d86525a281bc0250cef0b883b10034b not found: ID does not exist"
Apr 17 16:50:15.336627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.336595    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:50:15.341056 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.341033    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2vn4t"]
Apr 17 16:50:15.391034 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:50:15.390957    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" path="/var/lib/kubelet/pods/9fa73954-ebbb-4dde-8d14-8c964415beaf/volumes"
Apr 17 16:51:13.363359 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:51:13.363330    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:51:13.365727 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:51:13.365692    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:51:13.367528 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:51:13.367509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:51:13.369939 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:51:13.369920    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:52:29.789844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:29.789808    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerStarted","Data":"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"}
Apr 17 16:52:29.789844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:29.789842    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerStarted","Data":"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"}
Apr 17 16:52:29.790308 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:29.790017    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:52:29.817954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:29.817895    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" podStartSLOduration=6.428411943 podStartE2EDuration="2m19.817881229s" podCreationTimestamp="2026-04-17 16:50:10 +0000 UTC" firstStartedPulling="2026-04-17 16:50:15.280301396 +0000 UTC m=+1142.520959811" lastFinishedPulling="2026-04-17 16:52:28.66977067 +0000 UTC m=+1275.910429097" observedRunningTime="2026-04-17 16:52:29.816827271 +0000 UTC m=+1277.057485706" watchObservedRunningTime="2026-04-17 16:52:29.817881229 +0000 UTC m=+1277.058539664"
Apr 17 16:52:30.793561 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:30.793532    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:52:36.802196 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:52:36.802167    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:53:06.805831 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:06.805800    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:53:10.679446 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.679341    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:53:10.679947 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.679748    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kserve-container" containerID="cri-o://075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5" gracePeriod=30
Apr 17 16:53:10.680550 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.680129    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kube-rbac-proxy" containerID="cri-o://369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902" gracePeriod=30
Apr 17 16:53:10.792962 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.792928    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:10.793353 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793335    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="storage-initializer"
Apr 17 16:53:10.793443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="storage-initializer"
Apr 17 16:53:10.793443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793380    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kube-rbac-proxy"
Apr 17 16:53:10.793443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793389    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kube-rbac-proxy"
Apr 17 16:53:10.793443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793408    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container"
Apr 17 16:53:10.793443 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793417    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container"
Apr 17 16:53:10.793701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793499    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kserve-container"
Apr 17 16:53:10.793701 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.793518    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fa73954-ebbb-4dde-8d14-8c964415beaf" containerName="kube-rbac-proxy"
Apr 17 16:53:10.798795 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.798752    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.802224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.802202    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Apr 17 16:53:10.802350 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.802223    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Apr 17 16:53:10.808068 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.808046    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:10.884379 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.884344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.884379 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.884386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.884655 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.884455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.884655 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.884569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjz7\" (UniqueName: \"kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.938825 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.938728    2579 generic.go:358] "Generic (PLEG): container finished" podID="24a93556-738d-46ba-b2cc-984adcfb957d" containerID="369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902" exitCode=2
Apr 17 16:53:10.938825 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.938806    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerDied","Data":"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"}
Apr 17 16:53:10.985961 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.985922    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.986156 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.985974    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.986156 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.986022    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.986156 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:10.986091    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Apr 17 16:53:10.986156 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.986112    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjz7\" (UniqueName: \"kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.986362 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:10.986185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls podName:7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7 nodeName:}" failed. No retries permitted until 2026-04-17 16:53:11.486160353 +0000 UTC m=+1318.726818774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" (UID: "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Apr 17 16:53:10.986435 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.986369    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.986676 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.986657    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:10.995141 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:10.995115    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjz7\" (UniqueName: \"kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:11.492382 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.492343    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:11.494819 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.494788    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:11.710860 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.710830    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:11.736811 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.736785    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:53:11.794193 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.794131    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls\") pod \"24a93556-738d-46ba-b2cc-984adcfb957d\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") "
Apr 17 16:53:11.794333 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.794206    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location\") pod \"24a93556-738d-46ba-b2cc-984adcfb957d\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") "
Apr 17 16:53:11.794333 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.794259    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"24a93556-738d-46ba-b2cc-984adcfb957d\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") "
Apr 17 16:53:11.794333 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.794322    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhtck\" (UniqueName: \"kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck\") pod \"24a93556-738d-46ba-b2cc-984adcfb957d\" (UID: \"24a93556-738d-46ba-b2cc-984adcfb957d\") "
Apr 17 16:53:11.795081 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.795034    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24a93556-738d-46ba-b2cc-984adcfb957d" (UID: "24a93556-738d-46ba-b2cc-984adcfb957d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:53:11.795202 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.795117    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "24a93556-738d-46ba-b2cc-984adcfb957d" (UID: "24a93556-738d-46ba-b2cc-984adcfb957d"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:53:11.798208 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.798181    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck" (OuterVolumeSpecName: "kube-api-access-jhtck") pod "24a93556-738d-46ba-b2cc-984adcfb957d" (UID: "24a93556-738d-46ba-b2cc-984adcfb957d"). InnerVolumeSpecName "kube-api-access-jhtck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:53:11.799930 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.799909    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24a93556-738d-46ba-b2cc-984adcfb957d" (UID: "24a93556-738d-46ba-b2cc-984adcfb957d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:53:11.847251 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.847208    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:11.850116 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:53:11.850088    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdbdd9f_176d_4bf4_b3aa_89f4fcf1d4b7.slice/crio-fe1bf07bbd3cf27038fd9205c3dcc634e9cba4acd8b56db71136846b1edd85ef WatchSource:0}: Error finding container fe1bf07bbd3cf27038fd9205c3dcc634e9cba4acd8b56db71136846b1edd85ef: Status 404 returned error can't find the container with id fe1bf07bbd3cf27038fd9205c3dcc634e9cba4acd8b56db71136846b1edd85ef
Apr 17 16:53:11.895047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.895022    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a93556-738d-46ba-b2cc-984adcfb957d-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:11.895047 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.895047    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24a93556-738d-46ba-b2cc-984adcfb957d-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:11.895263 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.895058    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a93556-738d-46ba-b2cc-984adcfb957d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:11.895263 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.895074    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtck\" (UniqueName: \"kubernetes.io/projected/24a93556-738d-46ba-b2cc-984adcfb957d-kube-api-access-jhtck\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:11.943833 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.943801    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerStarted","Data":"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"}
Apr 17 16:53:11.944013 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.943840    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerStarted","Data":"fe1bf07bbd3cf27038fd9205c3dcc634e9cba4acd8b56db71136846b1edd85ef"}
Apr 17 16:53:11.945627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.945602    2579 generic.go:358] "Generic (PLEG): container finished" podID="24a93556-738d-46ba-b2cc-984adcfb957d" containerID="075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5" exitCode=0
Apr 17 16:53:11.945728 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.945649    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerDied","Data":"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"}
Apr 17 16:53:11.945728 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.945671    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf" event={"ID":"24a93556-738d-46ba-b2cc-984adcfb957d","Type":"ContainerDied","Data":"6f27fbcb833d265e9654053c9e32b51631b748330d445461e584baacd4869909"}
Apr 17 16:53:11.945728 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.945676    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"
Apr 17 16:53:11.945728 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.945689    2579 scope.go:117] "RemoveContainer" containerID="369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"
Apr 17 16:53:11.954805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.954784    2579 scope.go:117] "RemoveContainer" containerID="075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"
Apr 17 16:53:11.962809 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.962791    2579 scope.go:117] "RemoveContainer" containerID="f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"
Apr 17 16:53:11.969949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.969927    2579 scope.go:117] "RemoveContainer" containerID="369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"
Apr 17 16:53:11.970212 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:11.970193    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902\": container with ID starting with 369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902 not found: ID does not exist" containerID="369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"
Apr 17 16:53:11.970277 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.970219    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902"} err="failed to get container status \"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902\": rpc error: code = NotFound desc = could not find container \"369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902\": container with ID starting with 369594807a5fe0772f69c2f9804af25a74c05f23de49ec2a820c336a8cb6f902 not found: ID does not exist"
Apr 17 16:53:11.970277 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.970234    2579 scope.go:117] "RemoveContainer" containerID="075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"
Apr 17 16:53:11.970504 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:11.970420    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5\": container with ID starting with 075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5 not found: ID does not exist" containerID="075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"
Apr 17 16:53:11.970504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.970451    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5"} err="failed to get container status \"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5\": rpc error: code = NotFound desc = could not find container \"075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5\": container with ID starting with 075db37442d235d5041e06563c7923ad3fee85599752c78b8438db77140145f5 not found: ID does not exist"
Apr 17 16:53:11.970504 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.970475    2579 scope.go:117] "RemoveContainer" containerID="f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"
Apr 17 16:53:11.970700 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:11.970682    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148\": container with ID starting with f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148 not found: ID does not exist" containerID="f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"
Apr 17 16:53:11.970898 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.970702    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148"} err="failed to get container status \"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148\": rpc error: code = NotFound desc = could not find container \"f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148\": container with ID starting with f6b9e33d456743f984b792eecc3ceee368f071d31fc5970b963bab1582021148 not found: ID does not exist"
Apr 17 16:53:11.974358 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.974337    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:53:11.977565 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:11.977542    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-46kbf"]
Apr 17 16:53:13.390977 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:13.390946    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" path="/var/lib/kubelet/pods/24a93556-738d-46ba-b2cc-984adcfb957d/volumes"
Apr 17 16:53:15.961015 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:15.960976    2579 generic.go:358] "Generic (PLEG): container finished" podID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerID="f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3" exitCode=0
Apr 17 16:53:15.961392 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:15.961051    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerDied","Data":"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"}
Apr 17 16:53:16.966825 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.966792    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerStarted","Data":"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"}
Apr 17 16:53:16.966825 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.966831    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerStarted","Data":"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"}
Apr 17 16:53:16.967232 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.967129    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:16.967285 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.967269    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:16.968841 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.968812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 17 16:53:16.985732 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:16.985658    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podStartSLOduration=6.985644032 podStartE2EDuration="6.985644032s" podCreationTimestamp="2026-04-17 16:53:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:53:16.984374862 +0000 UTC m=+1324.225033310" watchObservedRunningTime="2026-04-17 16:53:16.985644032 +0000 UTC m=+1324.226302468"
Apr 17 16:53:17.970242 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:17.970199    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 17 16:53:22.974398 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:22.974363    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:22.974979 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:22.974951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 17 16:53:32.975496 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:32.975415    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:40.831239 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.831200    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:40.831800 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.831534    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container" containerID="cri-o://28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0" gracePeriod=30
Apr 17 16:53:40.831800 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.831564    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kube-rbac-proxy" containerID="cri-o://8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4" gracePeriod=30
Apr 17 16:53:40.935940 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.935906    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:53:40.936353 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936335    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="storage-initializer"
Apr 17 16:53:40.936353 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="storage-initializer"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936371    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kserve-container"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kserve-container"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936392    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kube-rbac-proxy"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936400    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kube-rbac-proxy"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936489    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kserve-container"
Apr 17 16:53:40.936538 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.936507    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="24a93556-738d-46ba-b2cc-984adcfb957d" containerName="kube-rbac-proxy"
Apr 17 16:53:40.940020 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.939998    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:40.942073 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.942051    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Apr 17 16:53:40.942073 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.942064    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 17 16:53:40.950330 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:40.950308    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:53:41.046895 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.046865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.047070 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.046910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.047070 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.046946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.047070 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.046963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxbx\" (UniqueName: \"kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.058739 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.058712    2579 generic.go:358] "Generic (PLEG): container finished" podID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerID="8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4" exitCode=2
Apr 17 16:53:41.058898 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.058792    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerDied","Data":"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"}
Apr 17 16:53:41.147623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.147574    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.147623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.147636    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.147896 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.147668    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.147896 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.147685    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxbx\" (UniqueName: \"kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.148166 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.148140    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.148484 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.148460    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.150177 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.150159    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.156068 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.156037    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxbx\" (UniqueName: \"kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.251985 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.251950    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:41.379800 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.379779    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:53:41.416617 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:53:41.416555    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b052066_a96a_4b4e_8c16_2e45ea20a2cd.slice/crio-ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01 WatchSource:0}: Error finding container ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01: Status 404 returned error can't find the container with id ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01
Apr 17 16:53:41.571043 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.571021    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:41.753147 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753062    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location\") pod \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") "
Apr 17 16:53:41.753147 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753130    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") "
Apr 17 16:53:41.753391 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753193    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") pod \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") "
Apr 17 16:53:41.753391 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753226    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjz7\" (UniqueName: \"kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7\") pod \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\" (UID: \"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7\") "
Apr 17 16:53:41.753509 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753482    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" (UID: "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:53:41.753562 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.753486    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" (UID: "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:53:41.755166 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.755145    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" (UID: "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:53:41.755357 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.755332    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7" (OuterVolumeSpecName: "kube-api-access-nvjz7") pod "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" (UID: "7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7"). InnerVolumeSpecName "kube-api-access-nvjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:53:41.853975 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.853947    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:41.853975 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.853976    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjz7\" (UniqueName: \"kubernetes.io/projected/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kube-api-access-nvjz7\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:41.854311 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.853988    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:41.854311 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:41.853998    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:53:42.063440 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.063344    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerStarted","Data":"3e68d51c9483a64c279db1ae032dcaeace08aedc4c8aa6a45281df7aac2c3a91"}
Apr 17 16:53:42.063440 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.063386    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerStarted","Data":"ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01"}
Apr 17 16:53:42.065190 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.065164    2579 generic.go:358] "Generic (PLEG): container finished" podID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerID="28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0" exitCode=0
Apr 17 16:53:42.065312 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.065203    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerDied","Data":"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"}
Apr 17 16:53:42.065312 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.065227    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"
Apr 17 16:53:42.065312 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.065237    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws" event={"ID":"7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7","Type":"ContainerDied","Data":"fe1bf07bbd3cf27038fd9205c3dcc634e9cba4acd8b56db71136846b1edd85ef"}
Apr 17 16:53:42.065312 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.065255    2579 scope.go:117] "RemoveContainer" containerID="8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"
Apr 17 16:53:42.079657 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.079617    2579 scope.go:117] "RemoveContainer" containerID="28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"
Apr 17 16:53:42.087724 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.087704    2579 scope.go:117] "RemoveContainer" containerID="f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"
Apr 17 16:53:42.091820 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.091800    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:42.096323 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096303    2579 scope.go:117] "RemoveContainer" containerID="8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"
Apr 17 16:53:42.096477 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096458    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t9xws"]
Apr 17 16:53:42.096582 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:42.096564    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4\": container with ID starting with 8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4 not found: ID does not exist" containerID="8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"
Apr 17 16:53:42.096634 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096591    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4"} err="failed to get container status \"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4\": rpc error: code = NotFound desc = could not find container \"8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4\": container with ID starting with 8306282e6cc6c068e473e4248ee82582dc1234c4fc1b936f5219cdfac9a9e6b4 not found: ID does not exist"
Apr 17 16:53:42.096634 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096610    2579 scope.go:117] "RemoveContainer" containerID="28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"
Apr 17 16:53:42.096862 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:42.096845    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0\": container with ID starting with 28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0 not found: ID does not exist" containerID="28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"
Apr 17 16:53:42.096920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0"} err="failed to get container status \"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0\": rpc error: code = NotFound desc = could not find container \"28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0\": container with ID starting with 28b0c134784d22d0601f40abffbf3c9c40c38276e92df9dd3da512f63fe5a5c0 not found: ID does not exist"
Apr 17 16:53:42.096920 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.096889    2579 scope.go:117] "RemoveContainer" containerID="f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"
Apr 17 16:53:42.097092 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:53:42.097077    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3\": container with ID starting with f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3 not found: ID does not exist" containerID="f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"
Apr 17 16:53:42.097138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:42.097096    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3"} err="failed to get container status \"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3\": rpc error: code = NotFound desc = could not find container \"f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3\": container with ID starting with f17ea6d21f5d9224619521bc5072d1d44a2671bf04c7b1a4e0a373f233f5c3c3 not found: ID does not exist"
Apr 17 16:53:43.389476 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:43.389437    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" path="/var/lib/kubelet/pods/7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7/volumes"
Apr 17 16:53:46.079661 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:46.079629    2579 generic.go:358] "Generic (PLEG): container finished" podID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerID="3e68d51c9483a64c279db1ae032dcaeace08aedc4c8aa6a45281df7aac2c3a91" exitCode=0
Apr 17 16:53:46.080061 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:46.079701    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerDied","Data":"3e68d51c9483a64c279db1ae032dcaeace08aedc4c8aa6a45281df7aac2c3a91"}
Apr 17 16:53:47.085342 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:47.085304    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerStarted","Data":"1f52ed90bdbe843cb1639c217346ea977e9f6ba97878735d1526d407e4127e8a"}
Apr 17 16:53:47.085342 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:47.085345    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerStarted","Data":"7511245a520d5b88485b6ce201c90e4d841e83a861ed792c66813aff0b35ee2a"}
Apr 17 16:53:47.085940 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:47.085634    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:47.085940 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:47.085664    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:53:47.104716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:47.104672    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" podStartSLOduration=7.10465686 podStartE2EDuration="7.10465686s" podCreationTimestamp="2026-04-17 16:53:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 16:53:47.103834235 +0000 UTC m=+1354.344492671" watchObservedRunningTime="2026-04-17 16:53:47.10465686 +0000 UTC m=+1354.345315297"
Apr 17 16:53:53.094527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:53:53.094495    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:54:23.098847 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:23.098817    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:54:31.005252 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.005214    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:54:31.005708 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.005629    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kserve-container" containerID="cri-o://7511245a520d5b88485b6ce201c90e4d841e83a861ed792c66813aff0b35ee2a" gracePeriod=30
Apr 17 16:54:31.005798 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.005695    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kube-rbac-proxy" containerID="cri-o://1f52ed90bdbe843cb1639c217346ea977e9f6ba97878735d1526d407e4127e8a" gracePeriod=30
Apr 17 16:54:31.090004 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.089939    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:54:31.090465 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090441    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container"
Apr 17 16:54:31.090589 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090468    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container"
Apr 17 16:54:31.090589 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090485    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="storage-initializer"
Apr 17 16:54:31.090589 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090493    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="storage-initializer"
Apr 17 16:54:31.090589 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090524    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kube-rbac-proxy"
Apr 17 16:54:31.090589 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090532    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kube-rbac-proxy"
Apr 17 16:54:31.090894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090645    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kserve-container"
Apr 17 16:54:31.090894 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.090660    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bdbdd9f-176d-4bf4-b3aa-89f4fcf1d4b7" containerName="kube-rbac-proxy"
Apr 17 16:54:31.094252 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.094229    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.096457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.096433    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Apr 17 16:54:31.096457 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.096448    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Apr 17 16:54:31.104859 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.104834    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:54:31.191318 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.191285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.191527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.191343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.191527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.191457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.191527 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.191513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk64s\" (UniqueName: \"kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.245141 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.245110    2579 generic.go:358] "Generic (PLEG): container finished" podID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerID="1f52ed90bdbe843cb1639c217346ea977e9f6ba97878735d1526d407e4127e8a" exitCode=2
Apr 17 16:54:31.245305 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.245181    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerDied","Data":"1f52ed90bdbe843cb1639c217346ea977e9f6ba97878735d1526d407e4127e8a"}
Apr 17 16:54:31.292245 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.292163    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xk64s\" (UniqueName: \"kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.292245 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.292205    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.292245 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.292237    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.292523 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.292286    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.292733 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.292709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.293059 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.293042    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.294884 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.294859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.299905 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.299884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk64s\" (UniqueName: \"kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s\") pod \"isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.406053 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.406020    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:31.533492 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:31.533461    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:54:31.536123 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:54:31.536097    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c7421d_22b0_475a_ad50_3943100cc496.slice/crio-201c4be0219d37a6320f1d86b4f3fd6f0310b0ff8ee2a3ab747a9d8de0b016a0 WatchSource:0}: Error finding container 201c4be0219d37a6320f1d86b4f3fd6f0310b0ff8ee2a3ab747a9d8de0b016a0: Status 404 returned error can't find the container with id 201c4be0219d37a6320f1d86b4f3fd6f0310b0ff8ee2a3ab747a9d8de0b016a0
Apr 17 16:54:32.250620 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.250594    2579 generic.go:358] "Generic (PLEG): container finished" podID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerID="7511245a520d5b88485b6ce201c90e4d841e83a861ed792c66813aff0b35ee2a" exitCode=0
Apr 17 16:54:32.250965 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.250669    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerDied","Data":"7511245a520d5b88485b6ce201c90e4d841e83a861ed792c66813aff0b35ee2a"}
Apr 17 16:54:32.250965 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.250708    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6" event={"ID":"7b052066-a96a-4b4e-8c16-2e45ea20a2cd","Type":"ContainerDied","Data":"ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01"}
Apr 17 16:54:32.250965 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.250724    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac03b48a01454c585e8ca9ea2872cf2b334d3425bec847bf4fd3b88db563de01"
Apr 17 16:54:32.252132 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.252104    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerStarted","Data":"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"}
Apr 17 16:54:32.252248 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.252138    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerStarted","Data":"201c4be0219d37a6320f1d86b4f3fd6f0310b0ff8ee2a3ab747a9d8de0b016a0"}
Apr 17 16:54:32.260427 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.260411    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:54:32.402513 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402484    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location\") pod \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") "
Apr 17 16:54:32.402682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402529    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls\") pod \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") "
Apr 17 16:54:32.402682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402658    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") "
Apr 17 16:54:32.402786 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402720    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbxbx\" (UniqueName: \"kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx\") pod \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\" (UID: \"7b052066-a96a-4b4e-8c16-2e45ea20a2cd\") "
Apr 17 16:54:32.402910 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402878    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b052066-a96a-4b4e-8c16-2e45ea20a2cd" (UID: "7b052066-a96a-4b4e-8c16-2e45ea20a2cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:54:32.402910 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.402903    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "7b052066-a96a-4b4e-8c16-2e45ea20a2cd" (UID: "7b052066-a96a-4b4e-8c16-2e45ea20a2cd"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:54:32.403103 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.403083    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:54:32.403164 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.403106    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:54:32.404646 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.404624    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b052066-a96a-4b4e-8c16-2e45ea20a2cd" (UID: "7b052066-a96a-4b4e-8c16-2e45ea20a2cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:54:32.404747 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.404733    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx" (OuterVolumeSpecName: "kube-api-access-wbxbx") pod "7b052066-a96a-4b4e-8c16-2e45ea20a2cd" (UID: "7b052066-a96a-4b4e-8c16-2e45ea20a2cd"). InnerVolumeSpecName "kube-api-access-wbxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:54:32.504506 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.504477    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:54:32.504506 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:32.504502    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxbx\" (UniqueName: \"kubernetes.io/projected/7b052066-a96a-4b4e-8c16-2e45ea20a2cd-kube-api-access-wbxbx\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:54:33.255524 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:33.255490    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"
Apr 17 16:54:33.277013 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:33.276986    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:54:33.280525 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:33.280494    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-znqb6"]
Apr 17 16:54:33.389407 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:33.389365    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" path="/var/lib/kubelet/pods/7b052066-a96a-4b4e-8c16-2e45ea20a2cd/volumes"
Apr 17 16:54:35.264200 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:35.264166    2579 generic.go:358] "Generic (PLEG): container finished" podID="f9c7421d-22b0-475a-ad50-3943100cc496" containerID="f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf" exitCode=0
Apr 17 16:54:35.264541 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:35.264236    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerDied","Data":"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"}
Apr 17 16:54:36.269031 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:36.268997    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerStarted","Data":"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"}
Apr 17 16:54:38.277949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.277911    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerStarted","Data":"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"}
Apr 17 16:54:38.277949 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.277951    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerStarted","Data":"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"}
Apr 17 16:54:38.278642 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.278058    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:38.278642 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.278153    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:38.278642 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.278182    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:54:38.298817 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:38.298739    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podStartSLOduration=4.8246710870000005 podStartE2EDuration="7.298726712s" podCreationTimestamp="2026-04-17 16:54:31 +0000 UTC" firstStartedPulling="2026-04-17 16:54:35.331256677 +0000 UTC m=+1402.571915091" lastFinishedPulling="2026-04-17 16:54:37.805312289 +0000 UTC m=+1405.045970716" observedRunningTime="2026-04-17 16:54:38.29743128 +0000 UTC m=+1405.538089719" watchObservedRunningTime="2026-04-17 16:54:38.298726712 +0000 UTC m=+1405.539385149"
Apr 17 16:54:44.287987 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:54:44.287955    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:55:14.290468 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:14.290435    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:55:44.291233 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:44.291203    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:55:51.186979 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.186944    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:55:51.187518 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.187389    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container" containerID="cri-o://fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e" gracePeriod=30
Apr 17 16:55:51.187604 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.187566    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" containerID="cri-o://47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3" gracePeriod=30
Apr 17 16:55:51.187682 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.187659    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-agent" containerID="cri-o://7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce" gracePeriod=30
Apr 17 16:55:51.532114 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.532031    2579 generic.go:358] "Generic (PLEG): container finished" podID="f9c7421d-22b0-475a-ad50-3943100cc496" containerID="47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3" exitCode=2
Apr 17 16:55:51.532262 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:51.532105    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerDied","Data":"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"}
Apr 17 16:55:53.541086 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:53.540992    2579 generic.go:358] "Generic (PLEG): container finished" podID="f9c7421d-22b0-475a-ad50-3943100cc496" containerID="fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e" exitCode=0
Apr 17 16:55:53.541086 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:53.541062    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerDied","Data":"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"}
Apr 17 16:55:54.282921 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:54.282881    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:55:54.289436 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:54.289409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 17 16:55:59.283323 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:55:59.283279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:56:04.282975 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:04.282938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:56:04.283346 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:04.283065    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:56:04.289275 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:04.289251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 17 16:56:09.282851 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:09.282813    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:56:13.393775 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:13.393729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:56:13.397297 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:13.397274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:56:13.397448 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:13.397345    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 16:56:13.400623 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:13.400607    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 16:56:14.282663 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:14.282621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:56:14.289383 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:14.289355    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 17 16:56:14.289490 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:14.289469    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:56:19.283035 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:19.282990    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 17 16:56:21.364910 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.364888    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:56:21.445560 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445481    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"f9c7421d-22b0-475a-ad50-3943100cc496\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") "
Apr 17 16:56:21.445560 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445546    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk64s\" (UniqueName: \"kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s\") pod \"f9c7421d-22b0-475a-ad50-3943100cc496\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") "
Apr 17 16:56:21.445832 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445589    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location\") pod \"f9c7421d-22b0-475a-ad50-3943100cc496\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") "
Apr 17 16:56:21.445832 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445618    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls\") pod \"f9c7421d-22b0-475a-ad50-3943100cc496\" (UID: \"f9c7421d-22b0-475a-ad50-3943100cc496\") "
Apr 17 16:56:21.445960 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445917    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "f9c7421d-22b0-475a-ad50-3943100cc496" (UID: "f9c7421d-22b0-475a-ad50-3943100cc496"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:56:21.446024 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.445957    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9c7421d-22b0-475a-ad50-3943100cc496" (UID: "f9c7421d-22b0-475a-ad50-3943100cc496"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:56:21.447649 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.447629    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9c7421d-22b0-475a-ad50-3943100cc496" (UID: "f9c7421d-22b0-475a-ad50-3943100cc496"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:56:21.447716 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.447694    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s" (OuterVolumeSpecName: "kube-api-access-xk64s") pod "f9c7421d-22b0-475a-ad50-3943100cc496" (UID: "f9c7421d-22b0-475a-ad50-3943100cc496"). InnerVolumeSpecName "kube-api-access-xk64s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:56:21.546823 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.546793    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c7421d-22b0-475a-ad50-3943100cc496-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:56:21.546823 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.546821    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c7421d-22b0-475a-ad50-3943100cc496-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:56:21.547012 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.546832    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c7421d-22b0-475a-ad50-3943100cc496-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:56:21.547012 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.546842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk64s\" (UniqueName: \"kubernetes.io/projected/f9c7421d-22b0-475a-ad50-3943100cc496-kube-api-access-xk64s\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:56:21.637495 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.637464    2579 generic.go:358] "Generic (PLEG): container finished" podID="f9c7421d-22b0-475a-ad50-3943100cc496" containerID="7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce" exitCode=137
Apr 17 16:56:21.637632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.637528    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerDied","Data":"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"}
Apr 17 16:56:21.637632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.637557    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr" event={"ID":"f9c7421d-22b0-475a-ad50-3943100cc496","Type":"ContainerDied","Data":"201c4be0219d37a6320f1d86b4f3fd6f0310b0ff8ee2a3ab747a9d8de0b016a0"}
Apr 17 16:56:21.637632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.637571    2579 scope.go:117] "RemoveContainer" containerID="47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"
Apr 17 16:56:21.637632 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.637577    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"
Apr 17 16:56:21.645818 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.645802    2579 scope.go:117] "RemoveContainer" containerID="7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"
Apr 17 16:56:21.653291 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.653267    2579 scope.go:117] "RemoveContainer" containerID="fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"
Apr 17 16:56:21.660629 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.660553    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:56:21.660688 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.660644    2579 scope.go:117] "RemoveContainer" containerID="f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"
Apr 17 16:56:21.665040 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.665019    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-5rclr"]
Apr 17 16:56:21.668849 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.668830    2579 scope.go:117] "RemoveContainer" containerID="47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"
Apr 17 16:56:21.669149 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:56:21.669133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3\": container with ID starting with 47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3 not found: ID does not exist" containerID="47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"
Apr 17 16:56:21.669191 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669158    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3"} err="failed to get container status \"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3\": rpc error: code = NotFound desc = could not find container \"47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3\": container with ID starting with 47a657143dbb6768962bfda7efd872fdd031966a24893deac280c1fadf93bcb3 not found: ID does not exist"
Apr 17 16:56:21.669191 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669178    2579 scope.go:117] "RemoveContainer" containerID="7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"
Apr 17 16:56:21.669429 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:56:21.669411    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce\": container with ID starting with 7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce not found: ID does not exist" containerID="7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"
Apr 17 16:56:21.669480 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669438    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce"} err="failed to get container status \"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce\": rpc error: code = NotFound desc = could not find container \"7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce\": container with ID starting with 7db2c39218377e36ae1349dd1c1961138da5f37c9c976146ad0bb687234010ce not found: ID does not exist"
Apr 17 16:56:21.669480 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669454    2579 scope.go:117] "RemoveContainer" containerID="fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"
Apr 17 16:56:21.669658 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:56:21.669644    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e\": container with ID starting with fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e not found: ID does not exist" containerID="fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"
Apr 17 16:56:21.669715 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669661    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e"} err="failed to get container status \"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e\": rpc error: code = NotFound desc = could not find container \"fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e\": container with ID starting with fe99d6cf2e06d3bf66b6f4ab8a3f838f3531c83aaa5d58a2def0a2eef59e6c7e not found: ID does not exist"
Apr 17 16:56:21.669715 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.669674    2579 scope.go:117] "RemoveContainer" containerID="f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"
Apr 17 16:56:21.670104 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:56:21.670087    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf\": container with ID starting with f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf not found: ID does not exist" containerID="f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"
Apr 17 16:56:21.670160 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:21.670119    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf"} err="failed to get container status \"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf\": rpc error: code = NotFound desc = could not find container \"f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf\": container with ID starting with f18a303aa7c7af8d7112d24a616687a41d189c5b8fa25eb137c7c5c5395e82bf not found: ID does not exist"
Apr 17 16:56:23.390562 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:56:23.390530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" path="/var/lib/kubelet/pods/f9c7421d-22b0-475a-ad50-3943100cc496/volumes"
Apr 17 16:57:12.896153 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896115    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="storage-initializer"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896449    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="storage-initializer"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896467    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="storage-initializer"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896472    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="storage-initializer"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896484    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kube-rbac-proxy"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kube-rbac-proxy"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896496    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-agent"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896502    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-agent"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896512    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896519    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896534    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896540    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896591    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896600    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kserve-agent"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896607    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kube-rbac-proxy"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896615    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b052066-a96a-4b4e-8c16-2e45ea20a2cd" containerName="kserve-container"
Apr 17 16:57:12.896665 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.896620    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9c7421d-22b0-475a-ad50-3943100cc496" containerName="kube-rbac-proxy"
Apr 17 16:57:12.899068 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.899045    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:12.905785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.901815    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Apr 17 16:57:12.905785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.902162    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 16:57:12.905785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.902448    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 16:57:12.905785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.902680    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Apr 17 16:57:12.905785 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.903004    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 16:57:12.911290 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:12.911260    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:57:13.000711 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.000678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r727c\" (UniqueName: \"kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.000711 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.000718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.000951 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.000745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.000951 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.000857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.101534 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.101495    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r727c\" (UniqueName: \"kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.101534 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.101534    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.101868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.101557    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.101868 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.101583    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.102020 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.101998    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.102261 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.102242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.104053 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.104033    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.109094 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.109074    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r727c\" (UniqueName: \"kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.216262 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.216161    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:13.344544 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.344517    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:57:13.347272 ip-10-0-141-140 kubenswrapper[2579]: W0417 16:57:13.347215    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3d51e0_6b56_4fda_8f96_55bc331126b7.slice/crio-70c95719020dfb91c634823073091cda0d720ef53bbcb306492347beb70bf9f2 WatchSource:0}: Error finding container 70c95719020dfb91c634823073091cda0d720ef53bbcb306492347beb70bf9f2: Status 404 returned error can't find the container with id 70c95719020dfb91c634823073091cda0d720ef53bbcb306492347beb70bf9f2
Apr 17 16:57:13.349283 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.349269    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 16:57:13.821892 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.821795    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerStarted","Data":"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"}
Apr 17 16:57:13.821892 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:13.821845    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerStarted","Data":"70c95719020dfb91c634823073091cda0d720ef53bbcb306492347beb70bf9f2"}
Apr 17 16:57:17.837518 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:17.837422    2579 generic.go:358] "Generic (PLEG): container finished" podID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerID="e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f" exitCode=0
Apr 17 16:57:17.837518 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:17.837497    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerDied","Data":"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"}
Apr 17 16:57:28.887064 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.887031    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerStarted","Data":"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"}
Apr 17 16:57:28.888709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.887074    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerStarted","Data":"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"}
Apr 17 16:57:28.888709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.887431    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:28.888709 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.887459    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:28.888869 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.888726    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:57:28.905844 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:28.905779    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podStartSLOduration=6.378993297 podStartE2EDuration="16.905744449s" podCreationTimestamp="2026-04-17 16:57:12 +0000 UTC" firstStartedPulling="2026-04-17 16:57:17.838697993 +0000 UTC m=+1565.079356406" lastFinishedPulling="2026-04-17 16:57:28.365449133 +0000 UTC m=+1575.606107558" observedRunningTime="2026-04-17 16:57:28.904502235 +0000 UTC m=+1576.145160671" watchObservedRunningTime="2026-04-17 16:57:28.905744449 +0000 UTC m=+1576.146402886"
Apr 17 16:57:29.891049 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:29.891016    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:57:34.896138 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:34.896109    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:57:34.896802 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:34.896747    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:57:44.897627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:44.897587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:57:54.896927 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:57:54.896885    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:58:04.897113 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:04.897068    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:58:14.898053 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:14.898022    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:58:24.402383 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:24.402347    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:58:24.403590 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:24.403530    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" containerID="cri-o://d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb" gracePeriod=30
Apr 17 16:58:24.403986 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:24.403925    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kube-rbac-proxy" containerID="cri-o://a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22" gracePeriod=30
Apr 17 16:58:24.891576 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:24.891533    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Apr 17 16:58:24.896927 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:24.896893    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 17 16:58:25.080321 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:25.080288    2579 generic.go:358] "Generic (PLEG): container finished" podID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerID="a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22" exitCode=2
Apr 17 16:58:25.080501 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:25.080359    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerDied","Data":"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"}
Apr 17 16:58:27.260794 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.260755    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:58:27.361330 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.361242    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r727c\" (UniqueName: \"kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c\") pod \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") "
Apr 17 16:58:27.361330 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.361301    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location\") pod \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") "
Apr 17 16:58:27.361330 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.361328    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls\") pod \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") "
Apr 17 16:58:27.361594 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.361374    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\" (UID: \"9c3d51e0-6b56-4fda-8f96-55bc331126b7\") "
Apr 17 16:58:27.361835 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.361802    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "9c3d51e0-6b56-4fda-8f96-55bc331126b7" (UID: "9c3d51e0-6b56-4fda-8f96-55bc331126b7"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 16:58:27.363460 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.363432    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c" (OuterVolumeSpecName: "kube-api-access-r727c") pod "9c3d51e0-6b56-4fda-8f96-55bc331126b7" (UID: "9c3d51e0-6b56-4fda-8f96-55bc331126b7"). InnerVolumeSpecName "kube-api-access-r727c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 16:58:27.363554 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.363511    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c3d51e0-6b56-4fda-8f96-55bc331126b7" (UID: "9c3d51e0-6b56-4fda-8f96-55bc331126b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 16:58:27.371287 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.371258    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c3d51e0-6b56-4fda-8f96-55bc331126b7" (UID: "9c3d51e0-6b56-4fda-8f96-55bc331126b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 16:58:27.462627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.462587    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r727c\" (UniqueName: \"kubernetes.io/projected/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kube-api-access-r727c\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:58:27.462627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.462621    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3d51e0-6b56-4fda-8f96-55bc331126b7-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:58:27.462627 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.462636    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3d51e0-6b56-4fda-8f96-55bc331126b7-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:58:27.462954 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:27.462652    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3d51e0-6b56-4fda-8f96-55bc331126b7-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 16:58:28.092003 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.091956    2579 generic.go:358] "Generic (PLEG): container finished" podID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerID="d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb" exitCode=0
Apr 17 16:58:28.092224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.092022    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerDied","Data":"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"}
Apr 17 16:58:28.092224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.092038    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"
Apr 17 16:58:28.092224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.092064    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n" event={"ID":"9c3d51e0-6b56-4fda-8f96-55bc331126b7","Type":"ContainerDied","Data":"70c95719020dfb91c634823073091cda0d720ef53bbcb306492347beb70bf9f2"}
Apr 17 16:58:28.092224 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.092080    2579 scope.go:117] "RemoveContainer" containerID="a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"
Apr 17 16:58:28.101559 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.101538    2579 scope.go:117] "RemoveContainer" containerID="d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"
Apr 17 16:58:28.109877 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.109851    2579 scope.go:117] "RemoveContainer" containerID="e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"
Apr 17 16:58:28.110684 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.110659    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:58:28.114295 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.114271    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-75r7n"]
Apr 17 16:58:28.118429 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.118403    2579 scope.go:117] "RemoveContainer" containerID="a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"
Apr 17 16:58:28.118729 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:58:28.118700    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22\": container with ID starting with a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22 not found: ID does not exist" containerID="a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"
Apr 17 16:58:28.118805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.118731    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22"} err="failed to get container status \"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22\": rpc error: code = NotFound desc = could not find container \"a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22\": container with ID starting with a7dee4e35f0b0539b1d524573aa35e81ed1ed35d7287f8ef25315dcc03afdb22 not found: ID does not exist"
Apr 17 16:58:28.118805 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.118751    2579 scope.go:117] "RemoveContainer" containerID="d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"
Apr 17 16:58:28.119012 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:58:28.118994    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb\": container with ID starting with d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb not found: ID does not exist" containerID="d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"
Apr 17 16:58:28.119059 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.119022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb"} err="failed to get container status \"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb\": rpc error: code = NotFound desc = could not find container \"d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb\": container with ID starting with d19de9b8de132cc09b77d16067017a23f18f68a77fdad639331cc6e5548ff9cb not found: ID does not exist"
Apr 17 16:58:28.119059 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.119039    2579 scope.go:117] "RemoveContainer" containerID="e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"
Apr 17 16:58:28.119277 ip-10-0-141-140 kubenswrapper[2579]: E0417 16:58:28.119259    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f\": container with ID starting with e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f not found: ID does not exist" containerID="e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"
Apr 17 16:58:28.119332 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:28.119284    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f"} err="failed to get container status \"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f\": rpc error: code = NotFound desc = could not find container \"e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f\": container with ID starting with e4e4317fdca595d2112d305813aa4a04f4579ba3eb09b3e65d41ff1fd631967f not found: ID does not exist"
Apr 17 16:58:29.389492 ip-10-0-141-140 kubenswrapper[2579]: I0417 16:58:29.389457    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" path="/var/lib/kubelet/pods/9c3d51e0-6b56-4fda-8f96-55bc331126b7/volumes"
Apr 17 17:00:13.494628 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:00:13.494597    2579 scope.go:117] "RemoveContainer" containerID="7511245a520d5b88485b6ce201c90e4d841e83a861ed792c66813aff0b35ee2a"
Apr 17 17:00:13.502495 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:00:13.502477    2579 scope.go:117] "RemoveContainer" containerID="3e68d51c9483a64c279db1ae032dcaeace08aedc4c8aa6a45281df7aac2c3a91"
Apr 17 17:00:13.509661 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:00:13.509639    2579 scope.go:117] "RemoveContainer" containerID="1f52ed90bdbe843cb1639c217346ea977e9f6ba97878735d1526d407e4127e8a"
Apr 17 17:01:13.417957 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:01:13.417916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:01:13.421548 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:01:13.421528    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:01:13.422365 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:01:13.422348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:01:13.425449 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:01:13.425434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:04:19.906656 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.906618    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.906976    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.906987    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907002    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="storage-initializer"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907008    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="storage-initializer"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kube-rbac-proxy"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907033    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kube-rbac-proxy"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907100    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kube-rbac-proxy"
Apr 17 17:04:19.907230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.907116    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c3d51e0-6b56-4fda-8f96-55bc331126b7" containerName="kserve-container"
Apr 17 17:04:19.911983 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.911952    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:19.915107 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.915074    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9efca4-predictor-serving-cert\""
Apr 17 17:04:19.915472 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.915446    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:04:19.915586 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.915472    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:04:19.915841 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.915822    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9efca4-kube-rbac-proxy-sar-config\""
Apr 17 17:04:19.915920 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.915863    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:04:19.920216 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.920195    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:04:19.992250 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.992214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7wx\" (UniqueName: \"kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:19.992440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.992268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:19.992440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.992359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:19.992440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:19.992418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.093558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.093515    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.093814 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.093606    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7wx\" (UniqueName: \"kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.093814 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.093639    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.093962 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.093916    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.093962 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.093951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.094821 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.094796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.096304 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.096269    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.101444 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.101425    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7wx\" (UniqueName: \"kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx\") pod \"isvc-primary-9efca4-predictor-897f6b668-4kf2v\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") " pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.228541 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.228461    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:20.356617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.356582    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:04:20.359295 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:04:20.359271    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffc4496_95db_4fe2_b6ce_b9160a8c0f33.slice/crio-fdec0c4e22ba0094ee4380162e3e35c277e6fb5d15ab057c5595057433c00a73 WatchSource:0}: Error finding container fdec0c4e22ba0094ee4380162e3e35c277e6fb5d15ab057c5595057433c00a73: Status 404 returned error can't find the container with id fdec0c4e22ba0094ee4380162e3e35c277e6fb5d15ab057c5595057433c00a73
Apr 17 17:04:20.361656 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:20.361637    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:04:21.275387 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:21.275349    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerStarted","Data":"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"}
Apr 17 17:04:21.275387 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:21.275391    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerStarted","Data":"fdec0c4e22ba0094ee4380162e3e35c277e6fb5d15ab057c5595057433c00a73"}
Apr 17 17:04:24.286859 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:24.286824    2579 generic.go:358] "Generic (PLEG): container finished" podID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerID="1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a" exitCode=0
Apr 17 17:04:24.287341 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:24.286898    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerDied","Data":"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"}
Apr 17 17:04:25.292464 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:25.292433    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerStarted","Data":"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"}
Apr 17 17:04:25.292464 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:25.292466    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerStarted","Data":"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"}
Apr 17 17:04:25.292915 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:25.292654    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:25.311358 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:25.311303    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podStartSLOduration=6.3112835480000005 podStartE2EDuration="6.311283548s" podCreationTimestamp="2026-04-17 17:04:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:04:25.308942441 +0000 UTC m=+1992.549600877" watchObservedRunningTime="2026-04-17 17:04:25.311283548 +0000 UTC m=+1992.551941985"
Apr 17 17:04:26.296932 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:26.296895    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:26.298174 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:26.298143    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:04:27.300715 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:27.300674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:04:32.305945 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:32.305913    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:04:32.306476 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:32.306449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:04:42.306938 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:42.306892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:04:52.307526 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:04:52.307486    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:05:02.307215 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:02.307172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:05:12.307338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:12.307300    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:05:22.307293 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:22.307249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:05:32.306962 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:32.306887    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:05:40.038256 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.038211    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:05:40.042185 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.042160    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.044380 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.044349    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9efca4\""
Apr 17 17:05:40.044539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.044355    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9efca4-predictor-serving-cert\""
Apr 17 17:05:40.044539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.044474    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\""
Apr 17 17:05:40.044539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.044505    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9efca4-dockercfg-8qtgj\""
Apr 17 17:05:40.044539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.044358    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 17 17:05:40.051434 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.051401    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:05:40.210243 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.210199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.210440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.210282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.210440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.210314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.210440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.210386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnbs\" (UniqueName: \"kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.210440 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.210434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.311812 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.311711    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnbs\" (UniqueName: \"kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.311812 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.311756    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.311812 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.311795    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.312070 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.311877    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.312070 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:05:40.311904    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-serving-cert: secret "isvc-secondary-9efca4-predictor-serving-cert" not found
Apr 17 17:05:40.312070 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.311917    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.312507 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:05:40.312388    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls podName:6f2d9e57-cc5f-4136-80d6-3a53a66d1f79 nodeName:}" failed. No retries permitted until 2026-04-17 17:05:40.812345935 +0000 UTC m=+2068.053004356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls") pod "isvc-secondary-9efca4-predictor-695c447fdc-c577g" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79") : secret "isvc-secondary-9efca4-predictor-serving-cert" not found
Apr 17 17:05:40.319000 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.312723    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.319000 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.313025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.319975 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.319942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.322059 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.322035    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnbs\" (UniqueName: \"kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.817228 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.817188    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.819851 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.819818    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") " pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:40.954608 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:40.954558    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:05:41.080044 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:41.079970    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:05:41.083505 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:05:41.083478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2d9e57_cc5f_4136_80d6_3a53a66d1f79.slice/crio-f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd WatchSource:0}: Error finding container f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd: Status 404 returned error can't find the container with id f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd
Apr 17 17:05:41.553244 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:41.553204    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerStarted","Data":"81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790"}
Apr 17 17:05:41.553244 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:41.553247    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerStarted","Data":"f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd"}
Apr 17 17:05:48.580876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:48.580846    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/0.log"
Apr 17 17:05:48.581253 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:48.580886    2579 generic.go:358] "Generic (PLEG): container finished" podID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerID="81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790" exitCode=1
Apr 17 17:05:48.581253 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:48.580970    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerDied","Data":"81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790"}
Apr 17 17:05:49.586462 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:49.586438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/0.log"
Apr 17 17:05:49.586847 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:49.586522    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerStarted","Data":"8942e5ea583d7e780cb4e37113e72d1c0e247bcc97c2cde626f29d2ce4d90c3d"}
Apr 17 17:05:51.594830 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.594724    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/1.log"
Apr 17 17:05:51.595188 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.595159    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/0.log"
Apr 17 17:05:51.595231 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.595189    2579 generic.go:358] "Generic (PLEG): container finished" podID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerID="8942e5ea583d7e780cb4e37113e72d1c0e247bcc97c2cde626f29d2ce4d90c3d" exitCode=1
Apr 17 17:05:51.595286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.595264    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerDied","Data":"8942e5ea583d7e780cb4e37113e72d1c0e247bcc97c2cde626f29d2ce4d90c3d"}
Apr 17 17:05:51.595323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.595309    2579 scope.go:117] "RemoveContainer" containerID="81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790"
Apr 17 17:05:51.595737 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:51.595711    2579 scope.go:117] "RemoveContainer" containerID="81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790"
Apr 17 17:05:51.607830 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:05:51.607797    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9efca4-predictor-695c447fdc-c577g_kserve-ci-e2e-test_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79_0 in pod sandbox f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd from index: no such id: '81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790'" containerID="81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790"
Apr 17 17:05:51.607929 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:05:51.607855    2579 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9efca4-predictor-695c447fdc-c577g_kserve-ci-e2e-test_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79_0 in pod sandbox f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd from index: no such id: '81aeea7a6706e20fd5764f3d9435554f4a27784baccef3ffc3c769dedd96b790'; Skipping pod \"isvc-secondary-9efca4-predictor-695c447fdc-c577g_kserve-ci-e2e-test(6f2d9e57-cc5f-4136-80d6-3a53a66d1f79)\"" logger="UnhandledError"
Apr 17 17:05:51.609212 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:05:51.609193    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9efca4-predictor-695c447fdc-c577g_kserve-ci-e2e-test(6f2d9e57-cc5f-4136-80d6-3a53a66d1f79)\"" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"
Apr 17 17:05:52.600333 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:05:52.600302    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/1.log"
Apr 17 17:06:00.111038 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.111008    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:06:00.155915 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.155884    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:06:00.156363 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.156304    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" containerID="cri-o://a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d" gracePeriod=30
Apr 17 17:06:00.156509 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.156349    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kube-rbac-proxy" containerID="cri-o://610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14" gracePeriod=30
Apr 17 17:06:00.261315 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.261288    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:00.266693 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.266672    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.269025 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.268896    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\""
Apr 17 17:06:00.269025 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.269013    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-96f18d-dockercfg-tg2nd\""
Apr 17 17:06:00.269376 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.269195    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-96f18d\""
Apr 17 17:06:00.269376 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.269258    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-96f18d-predictor-serving-cert\""
Apr 17 17:06:00.275902 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.275883    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:00.282036 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.282021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/1.log"
Apr 17 17:06:00.282144 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.282077    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:06:00.386663 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386628    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjnbs\" (UniqueName: \"kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs\") pod \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") "
Apr 17 17:06:00.386876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386675    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") pod \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") "
Apr 17 17:06:00.386876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386704    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert\") pod \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") "
Apr 17 17:06:00.386876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386720    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config\") pod \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") "
Apr 17 17:06:00.386876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386802    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location\") pod \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\" (UID: \"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79\") "
Apr 17 17:06:00.387036 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvbx\" (UniqueName: \"kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.387036 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.386983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.387036 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.387026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.387137 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.387058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.387171 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.387143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.390013 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.389991    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:06:00.390114 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.390054    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:06:00.390114 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.390097    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-9efca4-kube-rbac-proxy-sar-config") pod "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"). InnerVolumeSpecName "isvc-secondary-9efca4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:06:00.391869 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.391844    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs" (OuterVolumeSpecName: "kube-api-access-jjnbs") pod "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"). InnerVolumeSpecName "kube-api-access-jjnbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:06:00.391966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.391902    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" (UID: "6f2d9e57-cc5f-4136-80d6-3a53a66d1f79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:06:00.488493 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488454    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488503    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488533    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488567    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvbx\" (UniqueName: \"kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488582    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488620    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-cabundle-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488651    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-isvc-secondary-9efca4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488683    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:00.488697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488699    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnbs\" (UniqueName: \"kubernetes.io/projected/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-kube-api-access-jjnbs\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:00.489015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488716    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:00.489015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.488999    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.489310 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.489291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.489352 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.489292    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.491060 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.491041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.495814 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.495793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvbx\" (UniqueName: \"kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx\") pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.591833 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.591793    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:00.630810 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.630781    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9efca4-predictor-695c447fdc-c577g_6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/storage-initializer/1.log"
Apr 17 17:06:00.630977 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.630886    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g" event={"ID":"6f2d9e57-cc5f-4136-80d6-3a53a66d1f79","Type":"ContainerDied","Data":"f7047c6259410940e8e139d24b56782d854339992fe37bf763f5046bbae02bdd"}
Apr 17 17:06:00.630977 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.630932    2579 scope.go:117] "RemoveContainer" containerID="8942e5ea583d7e780cb4e37113e72d1c0e247bcc97c2cde626f29d2ce4d90c3d"
Apr 17 17:06:00.630977 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.630938    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"
Apr 17 17:06:00.633294 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.633268    2579 generic.go:358] "Generic (PLEG): container finished" podID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerID="610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14" exitCode=2
Apr 17 17:06:00.633419 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.633338    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerDied","Data":"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"}
Apr 17 17:06:00.670287 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.670261    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:06:00.674913 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.674884    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9efca4-predictor-695c447fdc-c577g"]
Apr 17 17:06:00.726959 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:00.726931    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:00.729734 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:06:00.729704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21420d1_4323_4c3b_8aba_8f118edbe2b5.slice/crio-6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0 WatchSource:0}: Error finding container 6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0: Status 404 returned error can't find the container with id 6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0
Apr 17 17:06:01.390069 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:01.390027    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" path="/var/lib/kubelet/pods/6f2d9e57-cc5f-4136-80d6-3a53a66d1f79/volumes"
Apr 17 17:06:01.640372 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:01.640285    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerStarted","Data":"f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504"}
Apr 17 17:06:01.640372 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:01.640319    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerStarted","Data":"6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0"}
Apr 17 17:06:02.301902 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:02.301863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Apr 17 17:06:02.306509 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:02.306484    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 17 17:06:04.402273 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.402249    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:06:04.523685 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.523603    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config\") pod \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") "
Apr 17 17:06:04.523685 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.523652    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw7wx\" (UniqueName: \"kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx\") pod \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") "
Apr 17 17:06:04.523896 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.523783    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls\") pod \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") "
Apr 17 17:06:04.523896 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.523848    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location\") pod \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\" (UID: \"dffc4496-95db-4fe2-b6ce-b9160a8c0f33\") "
Apr 17 17:06:04.524005 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.523981    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-9efca4-kube-rbac-proxy-sar-config") pod "dffc4496-95db-4fe2-b6ce-b9160a8c0f33" (UID: "dffc4496-95db-4fe2-b6ce-b9160a8c0f33"). InnerVolumeSpecName "isvc-primary-9efca4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:06:04.524191 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.524168    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-9efca4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-isvc-primary-9efca4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:04.524305 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.524202    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dffc4496-95db-4fe2-b6ce-b9160a8c0f33" (UID: "dffc4496-95db-4fe2-b6ce-b9160a8c0f33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:06:04.526299 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.526278    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dffc4496-95db-4fe2-b6ce-b9160a8c0f33" (UID: "dffc4496-95db-4fe2-b6ce-b9160a8c0f33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:06:04.526380 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.526299    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx" (OuterVolumeSpecName: "kube-api-access-zw7wx") pod "dffc4496-95db-4fe2-b6ce-b9160a8c0f33" (UID: "dffc4496-95db-4fe2-b6ce-b9160a8c0f33"). InnerVolumeSpecName "kube-api-access-zw7wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:06:04.624976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.624946    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:04.624976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.624971    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:04.624976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.624981    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7wx\" (UniqueName: \"kubernetes.io/projected/dffc4496-95db-4fe2-b6ce-b9160a8c0f33-kube-api-access-zw7wx\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:04.654228 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.654194    2579 generic.go:358] "Generic (PLEG): container finished" podID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerID="a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d" exitCode=0
Apr 17 17:06:04.654374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.654283    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"
Apr 17 17:06:04.654374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.654285    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerDied","Data":"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"}
Apr 17 17:06:04.654374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.654330    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v" event={"ID":"dffc4496-95db-4fe2-b6ce-b9160a8c0f33","Type":"ContainerDied","Data":"fdec0c4e22ba0094ee4380162e3e35c277e6fb5d15ab057c5595057433c00a73"}
Apr 17 17:06:04.654374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.654350    2579 scope.go:117] "RemoveContainer" containerID="610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"
Apr 17 17:06:04.664208 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.664185    2579 scope.go:117] "RemoveContainer" containerID="a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"
Apr 17 17:06:04.670872 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.670857    2579 scope.go:117] "RemoveContainer" containerID="1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"
Apr 17 17:06:04.677230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.677205    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:06:04.678784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.678750    2579 scope.go:117] "RemoveContainer" containerID="610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"
Apr 17 17:06:04.679079 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:04.679052    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14\": container with ID starting with 610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14 not found: ID does not exist" containerID="610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"
Apr 17 17:06:04.679177 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.679089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14"} err="failed to get container status \"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14\": rpc error: code = NotFound desc = could not find container \"610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14\": container with ID starting with 610a9ffb17e30f64b87efdb6071dd9539e3fb0f08d3bae92d41ad4f689695e14 not found: ID does not exist"
Apr 17 17:06:04.679177 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.679112    2579 scope.go:117] "RemoveContainer" containerID="a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"
Apr 17 17:06:04.679422 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:04.679383    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d\": container with ID starting with a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d not found: ID does not exist" containerID="a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"
Apr 17 17:06:04.679477 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.679431    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d"} err="failed to get container status \"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d\": rpc error: code = NotFound desc = could not find container \"a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d\": container with ID starting with a46da8a783037f443d904189200c82fdd5afeb1a0a1499fd2a6afc437d7e0e8d not found: ID does not exist"
Apr 17 17:06:04.679477 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.679467    2579 scope.go:117] "RemoveContainer" containerID="1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"
Apr 17 17:06:04.679720 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:04.679700    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a\": container with ID starting with 1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a not found: ID does not exist" containerID="1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"
Apr 17 17:06:04.679935 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.679721    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a"} err="failed to get container status \"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a\": rpc error: code = NotFound desc = could not find container \"1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a\": container with ID starting with 1fcdcd1c531ac150a929159479696c1040b7b7362b51a95775f4b70bce18757a not found: ID does not exist"
Apr 17 17:06:04.680430 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:04.680411    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9efca4-predictor-897f6b668-4kf2v"]
Apr 17 17:06:05.389730 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:05.389700    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" path="/var/lib/kubelet/pods/dffc4496-95db-4fe2-b6ce-b9160a8c0f33/volumes"
Apr 17 17:06:05.658644 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:05.658621    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/0.log"
Apr 17 17:06:05.659011 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:05.658657    2579 generic.go:358] "Generic (PLEG): container finished" podID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerID="f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504" exitCode=1
Apr 17 17:06:05.659011 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:05.658735    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerDied","Data":"f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504"}
Apr 17 17:06:06.664681 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:06.664652    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/0.log"
Apr 17 17:06:06.665097 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:06.664696    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerStarted","Data":"87455cef4d0c6d03b71c4f6a8c4b83e7bdad0c1507494b8ff014a6431dfaab72"}
Apr 17 17:06:08.672310 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.672238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/1.log"
Apr 17 17:06:08.672675 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.672559    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/0.log"
Apr 17 17:06:08.672675 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.672588    2579 generic.go:358] "Generic (PLEG): container finished" podID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerID="87455cef4d0c6d03b71c4f6a8c4b83e7bdad0c1507494b8ff014a6431dfaab72" exitCode=1
Apr 17 17:06:08.672747 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.672668    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerDied","Data":"87455cef4d0c6d03b71c4f6a8c4b83e7bdad0c1507494b8ff014a6431dfaab72"}
Apr 17 17:06:08.672747 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.672711    2579 scope.go:117] "RemoveContainer" containerID="f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504"
Apr 17 17:06:08.673119 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:08.673103    2579 scope.go:117] "RemoveContainer" containerID="f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504"
Apr 17 17:06:08.683310 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:08.683284    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_kserve-ci-e2e-test_c21420d1-4323-4c3b-8aba-8f118edbe2b5_0 in pod sandbox 6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0 from index: no such id: 'f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504'" containerID="f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504"
Apr 17 17:06:08.683377 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:08.683326    2579 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_kserve-ci-e2e-test_c21420d1-4323-4c3b-8aba-8f118edbe2b5_0 in pod sandbox 6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0 from index: no such id: 'f6bdad142a1a6d9975279a16684f325715ae7cb8fbcccd91bcd5408a4193b504'; Skipping pod \"isvc-init-fail-96f18d-predictor-78476558f5-zszdd_kserve-ci-e2e-test(c21420d1-4323-4c3b-8aba-8f118edbe2b5)\"" logger="UnhandledError"
Apr 17 17:06:08.684658 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:06:08.684638    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-96f18d-predictor-78476558f5-zszdd_kserve-ci-e2e-test(c21420d1-4323-4c3b-8aba-8f118edbe2b5)\"" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5"
Apr 17 17:06:09.677531 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:09.677503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/1.log"
Apr 17 17:06:10.244892 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.244860    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:10.397534 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.397511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/1.log"
Apr 17 17:06:10.397680 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.397577    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:10.478710 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.478681    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbvbx\" (UniqueName: \"kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx\") pod \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") "
Apr 17 17:06:10.478999 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.478734    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\") pod \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") "
Apr 17 17:06:10.478999 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.478785    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location\") pod \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") "
Apr 17 17:06:10.478999 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.478818    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls\") pod \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") "
Apr 17 17:06:10.478999 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.478891    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert\") pod \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\" (UID: \"c21420d1-4323-4c3b-8aba-8f118edbe2b5\") "
Apr 17 17:06:10.479219 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.479047    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c21420d1-4323-4c3b-8aba-8f118edbe2b5" (UID: "c21420d1-4323-4c3b-8aba-8f118edbe2b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:06:10.479219 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.479135    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:10.479219 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.479144    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-96f18d-kube-rbac-proxy-sar-config") pod "c21420d1-4323-4c3b-8aba-8f118edbe2b5" (UID: "c21420d1-4323-4c3b-8aba-8f118edbe2b5"). InnerVolumeSpecName "isvc-init-fail-96f18d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:06:10.479401 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.479335    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c21420d1-4323-4c3b-8aba-8f118edbe2b5" (UID: "c21420d1-4323-4c3b-8aba-8f118edbe2b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:06:10.480945 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.480922    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c21420d1-4323-4c3b-8aba-8f118edbe2b5" (UID: "c21420d1-4323-4c3b-8aba-8f118edbe2b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:06:10.481042 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.481018    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx" (OuterVolumeSpecName: "kube-api-access-mbvbx") pod "c21420d1-4323-4c3b-8aba-8f118edbe2b5" (UID: "c21420d1-4323-4c3b-8aba-8f118edbe2b5"). InnerVolumeSpecName "kube-api-access-mbvbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:06:10.580338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.580256    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-isvc-init-fail-96f18d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:10.580338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.580282    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21420d1-4323-4c3b-8aba-8f118edbe2b5-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:10.580338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.580292    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21420d1-4323-4c3b-8aba-8f118edbe2b5-cabundle-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:10.580338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.580301    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvbx\" (UniqueName: \"kubernetes.io/projected/c21420d1-4323-4c3b-8aba-8f118edbe2b5-kube-api-access-mbvbx\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:06:10.681875 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.681844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96f18d-predictor-78476558f5-zszdd_c21420d1-4323-4c3b-8aba-8f118edbe2b5/storage-initializer/1.log"
Apr 17 17:06:10.682265 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.681947    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd" event={"ID":"c21420d1-4323-4c3b-8aba-8f118edbe2b5","Type":"ContainerDied","Data":"6f0620a39abaf52cd966180baa98abf50d46f98ecbf7a44e04031509c3c08df0"}
Apr 17 17:06:10.682265 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.681970    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"
Apr 17 17:06:10.682265 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.682000    2579 scope.go:117] "RemoveContainer" containerID="87455cef4d0c6d03b71c4f6a8c4b83e7bdad0c1507494b8ff014a6431dfaab72"
Apr 17 17:06:10.739022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.738989    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:10.750909 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:10.750880    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96f18d-predictor-78476558f5-zszdd"]
Apr 17 17:06:11.389193 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:11.389163    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" path="/var/lib/kubelet/pods/c21420d1-4323-4c3b-8aba-8f118edbe2b5/volumes"
Apr 17 17:06:13.441916 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:13.441880    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:06:13.445582 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:13.445548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:06:13.446844 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:13.446822    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:06:13.450352 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:06:13.450333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:11:13.465276 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:11:13.465239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:11:13.468709 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:11:13.468667    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:11:13.471184 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:11:13.471163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:11:13.474591 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:11:13.474573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:15:23.928520 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.928484    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.928970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.928985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.928999    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="storage-initializer"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929007    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="storage-initializer"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929023    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kube-rbac-proxy"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929032    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kube-rbac-proxy"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929050    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929071    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.929084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929090    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929098    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929112    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929120    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929185    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929199    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929209    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kserve-container"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929221    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c21420d1-4323-4c3b-8aba-8f118edbe2b5" containerName="storage-initializer"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929230    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="dffc4496-95db-4fe2-b6ce-b9160a8c0f33" containerName="kube-rbac-proxy"
Apr 17 17:15:23.929412 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.929368    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="6f2d9e57-cc5f-4136-80d6-3a53a66d1f79" containerName="storage-initializer"
Apr 17 17:15:23.932952 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.932932    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:23.935095 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.935073    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Apr 17 17:15:23.935443 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.935415    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:15:23.935569 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.935491    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:15:23.936902 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.936871    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Apr 17 17:15:23.937087 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.937069    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:15:23.939138 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.939120    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:15:23.998067 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.998030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:23.998067 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.998068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:23.998273 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.998179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:23.998273 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:23.998209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47hb\" (UniqueName: \"kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099082 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099047    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099082 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099083    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099342 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099140    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099342 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099159    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k47hb\" (UniqueName: \"kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099560 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099539    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.099819 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.099798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.101614 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.101597    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.106944 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.106923    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47hb\" (UniqueName: \"kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb\") pod \"isvc-sklearn-predictor-d8dbfbbb9-xgzx7\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.244391 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.244294    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:24.364734 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.364710    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:15:24.367196 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:15:24.367170    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf020fda3_0a7c_41a8_a5ee_a44f93f37150.slice/crio-db87603964cd794809183eeaedd5dea53e70aa7f6bbdac9bb46caedb35ba4a35 WatchSource:0}: Error finding container db87603964cd794809183eeaedd5dea53e70aa7f6bbdac9bb46caedb35ba4a35: Status 404 returned error can't find the container with id db87603964cd794809183eeaedd5dea53e70aa7f6bbdac9bb46caedb35ba4a35
Apr 17 17:15:24.369144 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.369127    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:15:24.494593 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.494507    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerStarted","Data":"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"}
Apr 17 17:15:24.494593 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:24.494555    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerStarted","Data":"db87603964cd794809183eeaedd5dea53e70aa7f6bbdac9bb46caedb35ba4a35"}
Apr 17 17:15:28.508139 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:28.508106    2579 generic.go:358] "Generic (PLEG): container finished" podID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerID="b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5" exitCode=0
Apr 17 17:15:28.508619 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:28.508168    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerDied","Data":"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"}
Apr 17 17:15:29.512971 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:29.512938    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerStarted","Data":"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"}
Apr 17 17:15:29.512971 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:29.512970    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerStarted","Data":"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"}
Apr 17 17:15:29.513500 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:29.513162    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:29.532505 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:29.532455    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podStartSLOduration=6.532442455 podStartE2EDuration="6.532442455s" podCreationTimestamp="2026-04-17 17:15:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:15:29.531677237 +0000 UTC m=+2656.772335673" watchObservedRunningTime="2026-04-17 17:15:29.532442455 +0000 UTC m=+2656.773100890"
Apr 17 17:15:30.516209 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:30.516119    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:30.517681 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:30.517650    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:15:31.520045 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:31.520007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:15:36.524726 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:36.524698    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:15:36.525222 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:36.525194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:15:46.525221 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:46.525180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:15:56.525625 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:15:56.525543    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:16:06.525349 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:06.525311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:16:13.489697 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:13.489671    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:16:13.495975 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:13.495948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:16:13.498812 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:13.498788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:16:13.501967 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:13.501950    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:16:16.525895 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:16.525859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:16:26.526137 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:26.526088    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:16:36.525926 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:36.525892    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:16:44.038178 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.038145    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:16:44.038750 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.038465    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" containerID="cri-o://c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6" gracePeriod=30
Apr 17 17:16:44.038750 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.038506    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kube-rbac-proxy" containerID="cri-o://b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0" gracePeriod=30
Apr 17 17:16:44.118641 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.118609    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:16:44.128434 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.128402    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.130751 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.130570    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Apr 17 17:16:44.130751 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.130656    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 17 17:16:44.130751 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.130570    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:16:44.172347 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.172320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.172507 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.172371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrlr\" (UniqueName: \"kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.172507 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.172434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.172507 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.172478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273117 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273076    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273340 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273136    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273340 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273164    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrlr\" (UniqueName: \"kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273340 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273195    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273340 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:16:44.273313    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Apr 17 17:16:44.273575 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:16:44.273382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls podName:0e86bf4f-7c65-47ef-a11c-79a4d2b04098 nodeName:}" failed. No retries permitted until 2026-04-17 17:16:44.773366851 +0000 UTC m=+2732.014025270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls") pod "sklearn-v2-mlserver-predictor-65d8664766-nv54z" (UID: "0e86bf4f-7c65-47ef-a11c-79a4d2b04098") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Apr 17 17:16:44.273575 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273544    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.273815 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.273796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.284308 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.284282    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrlr\" (UniqueName: \"kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.773293 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.773258    2579 generic.go:358] "Generic (PLEG): container finished" podID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerID="b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0" exitCode=2
Apr 17 17:16:44.773451 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.773338    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerDied","Data":"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"}
Apr 17 17:16:44.778664 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.778638    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:44.781024 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:44.781009    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-nv54z\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:45.041906 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:45.041822    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:45.165456 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:45.165425    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:16:45.168927 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:16:45.168898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86bf4f_7c65_47ef_a11c_79a4d2b04098.slice/crio-997b947bebe1864be0504221b82356cb25d781357a395f9b820a257c49289097 WatchSource:0}: Error finding container 997b947bebe1864be0504221b82356cb25d781357a395f9b820a257c49289097: Status 404 returned error can't find the container with id 997b947bebe1864be0504221b82356cb25d781357a395f9b820a257c49289097
Apr 17 17:16:45.778505 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:45.778470    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerStarted","Data":"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"}
Apr 17 17:16:45.778505 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:45.778504    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerStarted","Data":"997b947bebe1864be0504221b82356cb25d781357a395f9b820a257c49289097"}
Apr 17 17:16:46.520950 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:46.520908    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Apr 17 17:16:46.525311 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:46.525287    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Apr 17 17:16:48.281032 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.281009    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:16:48.410101 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410020    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k47hb\" (UniqueName: \"kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb\") pod \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") "
Apr 17 17:16:48.410254 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410107    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location\") pod \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") "
Apr 17 17:16:48.410254 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410150    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls\") pod \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") "
Apr 17 17:16:48.410254 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410176    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\" (UID: \"f020fda3-0a7c-41a8-a5ee-a44f93f37150\") "
Apr 17 17:16:48.410498 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410463    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f020fda3-0a7c-41a8-a5ee-a44f93f37150" (UID: "f020fda3-0a7c-41a8-a5ee-a44f93f37150"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:16:48.410622 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.410537    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "f020fda3-0a7c-41a8-a5ee-a44f93f37150" (UID: "f020fda3-0a7c-41a8-a5ee-a44f93f37150"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:16:48.412199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.412169    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f020fda3-0a7c-41a8-a5ee-a44f93f37150" (UID: "f020fda3-0a7c-41a8-a5ee-a44f93f37150"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:16:48.412271 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.412170    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb" (OuterVolumeSpecName: "kube-api-access-k47hb") pod "f020fda3-0a7c-41a8-a5ee-a44f93f37150" (UID: "f020fda3-0a7c-41a8-a5ee-a44f93f37150"). InnerVolumeSpecName "kube-api-access-k47hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:16:48.511684 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.511652    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f020fda3-0a7c-41a8-a5ee-a44f93f37150-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:16:48.511684 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.511679    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f020fda3-0a7c-41a8-a5ee-a44f93f37150-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:16:48.511684 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.511689    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47hb\" (UniqueName: \"kubernetes.io/projected/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kube-api-access-k47hb\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:16:48.511926 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.511699    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f020fda3-0a7c-41a8-a5ee-a44f93f37150-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:16:48.790715 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.790633    2579 generic.go:358] "Generic (PLEG): container finished" podID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerID="c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6" exitCode=0
Apr 17 17:16:48.790882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.790712    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"
Apr 17 17:16:48.790882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.790719    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerDied","Data":"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"}
Apr 17 17:16:48.790882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.790756    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7" event={"ID":"f020fda3-0a7c-41a8-a5ee-a44f93f37150","Type":"ContainerDied","Data":"db87603964cd794809183eeaedd5dea53e70aa7f6bbdac9bb46caedb35ba4a35"}
Apr 17 17:16:48.790882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.790800    2579 scope.go:117] "RemoveContainer" containerID="b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"
Apr 17 17:16:48.801249 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.801225    2579 scope.go:117] "RemoveContainer" containerID="c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"
Apr 17 17:16:48.808382 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.808363    2579 scope.go:117] "RemoveContainer" containerID="b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"
Apr 17 17:16:48.812473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.812450    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:16:48.816913 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.816892    2579 scope.go:117] "RemoveContainer" containerID="b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"
Apr 17 17:16:48.817198 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:16:48.817168    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0\": container with ID starting with b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0 not found: ID does not exist" containerID="b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"
Apr 17 17:16:48.817304 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0"} err="failed to get container status \"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0\": rpc error: code = NotFound desc = could not find container \"b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0\": container with ID starting with b30a5dec603344a48f72c65676dd257b2eafe498a6116deffa5d379f2925f7a0 not found: ID does not exist"
Apr 17 17:16:48.817304 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817225    2579 scope.go:117] "RemoveContainer" containerID="c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"
Apr 17 17:16:48.817491 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:16:48.817470    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6\": container with ID starting with c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6 not found: ID does not exist" containerID="c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"
Apr 17 17:16:48.817553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817502    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-xgzx7"]
Apr 17 17:16:48.817553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817501    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6"} err="failed to get container status \"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6\": rpc error: code = NotFound desc = could not find container \"c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6\": container with ID starting with c05aa6146678c37ae259cf6b8002811eb71b3f90348e84874c0d15c3a035e5f6 not found: ID does not exist"
Apr 17 17:16:48.817553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817546    2579 scope.go:117] "RemoveContainer" containerID="b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"
Apr 17 17:16:48.817893 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:16:48.817866    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5\": container with ID starting with b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5 not found: ID does not exist" containerID="b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"
Apr 17 17:16:48.817993 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:48.817898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5"} err="failed to get container status \"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5\": rpc error: code = NotFound desc = could not find container \"b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5\": container with ID starting with b886ab993f8564277f91e4a80ac358f0112bb21c1a9cad650d7ba9fd5bf188e5 not found: ID does not exist"
Apr 17 17:16:49.389553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:49.389519    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" path="/var/lib/kubelet/pods/f020fda3-0a7c-41a8-a5ee-a44f93f37150/volumes"
Apr 17 17:16:49.796246 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:49.796163    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerID="c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0" exitCode=0
Apr 17 17:16:49.796246 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:49.796211    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerDied","Data":"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"}
Apr 17 17:16:50.801557 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:50.801514    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerStarted","Data":"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"}
Apr 17 17:16:50.801557 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:50.801560    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerStarted","Data":"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"}
Apr 17 17:16:50.802140 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:50.801837    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:50.821151 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:50.821090    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" podStartSLOduration=6.821069239 podStartE2EDuration="6.821069239s" podCreationTimestamp="2026-04-17 17:16:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:16:50.820536375 +0000 UTC m=+2738.061194813" watchObservedRunningTime="2026-04-17 17:16:50.821069239 +0000 UTC m=+2738.061727673"
Apr 17 17:16:51.805180 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:51.805148    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:16:57.814398 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:16:57.814368    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:17:27.818455 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:27.818362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 17 17:17:37.817384 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:37.817351    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:17:44.244812 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.244728    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:17:44.245244 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.245190    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kserve-container" containerID="cri-o://bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d" gracePeriod=30
Apr 17 17:17:44.245340 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.245270    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kube-rbac-proxy" containerID="cri-o://b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d" gracePeriod=30
Apr 17 17:17:44.322370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322329    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:17:44.322808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322790    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kube-rbac-proxy"
Apr 17 17:17:44.322917 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322811    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kube-rbac-proxy"
Apr 17 17:17:44.322917 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322844    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container"
Apr 17 17:17:44.322917 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322853    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container"
Apr 17 17:17:44.322917 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322870    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="storage-initializer"
Apr 17 17:17:44.322917 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322879    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="storage-initializer"
Apr 17 17:17:44.323176 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322966    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kube-rbac-proxy"
Apr 17 17:17:44.323176 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.322986    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f020fda3-0a7c-41a8-a5ee-a44f93f37150" containerName="kserve-container"
Apr 17 17:17:44.326646 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.326624    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.329415 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.329384    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Apr 17 17:17:44.329546 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.329419    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Apr 17 17:17:44.337903 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.337877    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:17:44.405183 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.405148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.405375 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.405194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2dp\" (UniqueName: \"kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.405375 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.405224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.405375 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.405258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.506558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.506453    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.506558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.506518    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2dp\" (UniqueName: \"kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.506558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.506552    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.506941 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.506600    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.506941 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:17:44.506794    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Apr 17 17:17:44.506941 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:17:44.506875    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls podName:9d96031c-66ad-492d-97a9-33caa5d83443 nodeName:}" failed. No retries permitted until 2026-04-17 17:17:45.006852796 +0000 UTC m=+2792.247511216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls") pod "isvc-sklearn-runtime-predictor-65cd49579f-pgv28" (UID: "9d96031c-66ad-492d-97a9-33caa5d83443") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Apr 17 17:17:44.507083 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.507015    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.507372 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.507347    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.517085 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.517053    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2dp\" (UniqueName: \"kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:44.986149 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.986115    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerID="b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d" exitCode=2
Apr 17 17:17:44.986300 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:44.986179    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerDied","Data":"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"}
Apr 17 17:17:45.010607 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.010561    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:45.013027 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.012997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65cd49579f-pgv28\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:45.240997 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.240905    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:45.369785 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.369738    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:17:45.372882 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:17:45.372851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96031c_66ad_492d_97a9_33caa5d83443.slice/crio-938fb7bdbe91be47e09b43b2d1c3af41c2c7c8d5c2195956610ccc644bb649e2 WatchSource:0}: Error finding container 938fb7bdbe91be47e09b43b2d1c3af41c2c7c8d5c2195956610ccc644bb649e2: Status 404 returned error can't find the container with id 938fb7bdbe91be47e09b43b2d1c3af41c2c7c8d5c2195956610ccc644bb649e2
Apr 17 17:17:45.990996 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.990962    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerStarted","Data":"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"}
Apr 17 17:17:45.990996 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:45.991000    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerStarted","Data":"938fb7bdbe91be47e09b43b2d1c3af41c2c7c8d5c2195956610ccc644bb649e2"}
Apr 17 17:17:47.809872 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:47.809833    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Apr 17 17:17:51.012115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.012080    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d96031c-66ad-492d-97a9-33caa5d83443" containerID="d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821" exitCode=0
Apr 17 17:17:51.012545 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.012154    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerDied","Data":"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"}
Apr 17 17:17:51.885368 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.885338    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:17:51.976115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976017    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") pod \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") "
Apr 17 17:17:51.976115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976067    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") "
Apr 17 17:17:51.976115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976094    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location\") pod \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") "
Apr 17 17:17:51.976411 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976201    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqrlr\" (UniqueName: \"kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr\") pod \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\" (UID: \"0e86bf4f-7c65-47ef-a11c-79a4d2b04098\") "
Apr 17 17:17:51.976492 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976455    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "0e86bf4f-7c65-47ef-a11c-79a4d2b04098" (UID: "0e86bf4f-7c65-47ef-a11c-79a4d2b04098"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:17:51.976567 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.976539    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e86bf4f-7c65-47ef-a11c-79a4d2b04098" (UID: "0e86bf4f-7c65-47ef-a11c-79a4d2b04098"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:17:51.978213 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.978186    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e86bf4f-7c65-47ef-a11c-79a4d2b04098" (UID: "0e86bf4f-7c65-47ef-a11c-79a4d2b04098"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:17:51.978384 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:51.978362    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr" (OuterVolumeSpecName: "kube-api-access-tqrlr") pod "0e86bf4f-7c65-47ef-a11c-79a4d2b04098" (UID: "0e86bf4f-7c65-47ef-a11c-79a4d2b04098"). InnerVolumeSpecName "kube-api-access-tqrlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:17:52.017813 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.017753    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerStarted","Data":"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"}
Apr 17 17:17:52.017813 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.017817    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerStarted","Data":"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"}
Apr 17 17:17:52.018364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.018138    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:52.018364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.018182    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:52.019602 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.019575    2579 generic.go:358] "Generic (PLEG): container finished" podID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerID="bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d" exitCode=0
Apr 17 17:17:52.019784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.019630    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerDied","Data":"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"}
Apr 17 17:17:52.019784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.019654    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z" event={"ID":"0e86bf4f-7c65-47ef-a11c-79a4d2b04098","Type":"ContainerDied","Data":"997b947bebe1864be0504221b82356cb25d781357a395f9b820a257c49289097"}
Apr 17 17:17:52.019784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.019671    2579 scope.go:117] "RemoveContainer" containerID="b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"
Apr 17 17:17:52.019784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.019676    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"
Apr 17 17:17:52.020023 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.020002    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Apr 17 17:17:52.028945 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.028926    2579 scope.go:117] "RemoveContainer" containerID="bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"
Apr 17 17:17:52.035935 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.035721    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podStartSLOduration=8.03570214 podStartE2EDuration="8.03570214s" podCreationTimestamp="2026-04-17 17:17:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:17:52.035453602 +0000 UTC m=+2799.276112039" watchObservedRunningTime="2026-04-17 17:17:52.03570214 +0000 UTC m=+2799.276360577"
Apr 17 17:17:52.038714 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.038685    2579 scope.go:117] "RemoveContainer" containerID="c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"
Apr 17 17:17:52.047276 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.047258    2579 scope.go:117] "RemoveContainer" containerID="b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"
Apr 17 17:17:52.047553 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:17:52.047536    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d\": container with ID starting with b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d not found: ID does not exist" containerID="b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"
Apr 17 17:17:52.047617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.047565    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d"} err="failed to get container status \"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d\": rpc error: code = NotFound desc = could not find container \"b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d\": container with ID starting with b2ec0bc3c8c8a4e956112ec20919c15e6c9c3e743dd939aa63789765cd3e566d not found: ID does not exist"
Apr 17 17:17:52.047617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.047584    2579 scope.go:117] "RemoveContainer" containerID="bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"
Apr 17 17:17:52.047843 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:17:52.047827    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d\": container with ID starting with bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d not found: ID does not exist" containerID="bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"
Apr 17 17:17:52.047910 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.047848    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d"} err="failed to get container status \"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d\": rpc error: code = NotFound desc = could not find container \"bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d\": container with ID starting with bf6132105ea01cd2c084ab36bcbbcf1554c059cf0cf0bfae7b4d993b0a42ae6d not found: ID does not exist"
Apr 17 17:17:52.047910 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.047868    2579 scope.go:117] "RemoveContainer" containerID="c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"
Apr 17 17:17:52.048086 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:17:52.048066    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0\": container with ID starting with c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0 not found: ID does not exist" containerID="c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"
Apr 17 17:17:52.048146 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.048097    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0"} err="failed to get container status \"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0\": rpc error: code = NotFound desc = could not find container \"c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0\": container with ID starting with c430c23c3dbed63adfb7d623fa49d332449188b1153612ae044aae7426ee98a0 not found: ID does not exist"
Apr 17 17:17:52.051608 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.051582    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:17:52.056533 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.056508    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-nv54z"]
Apr 17 17:17:52.076989 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.076944    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrlr\" (UniqueName: \"kubernetes.io/projected/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kube-api-access-tqrlr\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:17:52.076989 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.076974    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:17:52.076989 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.076990    2579 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:17:52.077305 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:52.077005    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e86bf4f-7c65-47ef-a11c-79a4d2b04098-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:17:53.024613 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:53.024577    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Apr 17 17:17:53.391275 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:53.391234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" path="/var/lib/kubelet/pods/0e86bf4f-7c65-47ef-a11c-79a4d2b04098/volumes"
Apr 17 17:17:58.030129 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:58.030098    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:17:58.030692 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:17:58.030664    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Apr 17 17:18:08.030966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:08.030925    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:18:21.304375 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.304338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-65cd49579f-pgv28_9d96031c-66ad-492d-97a9-33caa5d83443/kserve-container/0.log"
Apr 17 17:18:21.472682 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.472644    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:18:21.473200 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.473160    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container" containerID="cri-o://e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55" gracePeriod=30
Apr 17 17:18:21.473484 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.473178    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kube-rbac-proxy" containerID="cri-o://f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5" gracePeriod=30
Apr 17 17:18:21.565523 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.565430    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:18:21.565939 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.565921    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="storage-initializer"
Apr 17 17:18:21.566068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.565942    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="storage-initializer"
Apr 17 17:18:21.566068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.565975    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kserve-container"
Apr 17 17:18:21.566068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.565984    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kserve-container"
Apr 17 17:18:21.566068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.566009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kube-rbac-proxy"
Apr 17 17:18:21.566068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.566018    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kube-rbac-proxy"
Apr 17 17:18:21.566345 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.566118    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kserve-container"
Apr 17 17:18:21.566345 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.566140    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e86bf4f-7c65-47ef-a11c-79a4d2b04098" containerName="kube-rbac-proxy"
Apr 17 17:18:21.569556 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.569536    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.571881 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.571856    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Apr 17 17:18:21.572161 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.572139    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 17 17:18:21.580394 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.580233    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:18:21.644410 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.644371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.644592 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.644520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.644592 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.644560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvd7\" (UniqueName: \"kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.644726 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.644592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745181 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745142    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745391 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745241    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745391 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745278    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvd7\" (UniqueName: \"kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745391 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745314    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745622 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.745975 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.745957    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.747640 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.747623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.754378 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.754351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvd7\" (UniqueName: \"kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:21.882628 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:21.882596    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:22.014421 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.014392    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:18:22.017265 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:18:22.017234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d51f672_715a_49c1_aaf6_b7608482442c.slice/crio-61feccf5318b47f522c98fb5b6594a44da214bcd86efa43495378d2005a5c2a8 WatchSource:0}: Error finding container 61feccf5318b47f522c98fb5b6594a44da214bcd86efa43495378d2005a5c2a8: Status 404 returned error can't find the container with id 61feccf5318b47f522c98fb5b6594a44da214bcd86efa43495378d2005a5c2a8
Apr 17 17:18:22.130567 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.130532    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d96031c-66ad-492d-97a9-33caa5d83443" containerID="f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5" exitCode=2
Apr 17 17:18:22.130786 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.130611    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerDied","Data":"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"}
Apr 17 17:18:22.132054 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.132012    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerStarted","Data":"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"}
Apr 17 17:18:22.132169 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.132065    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerStarted","Data":"61feccf5318b47f522c98fb5b6594a44da214bcd86efa43495378d2005a5c2a8"}
Apr 17 17:18:22.533119 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.533094    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:18:22.654387 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.654337    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"9d96031c-66ad-492d-97a9-33caa5d83443\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") "
Apr 17 17:18:22.654590 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.654450    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location\") pod \"9d96031c-66ad-492d-97a9-33caa5d83443\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") "
Apr 17 17:18:22.654590 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.654471    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd2dp\" (UniqueName: \"kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp\") pod \"9d96031c-66ad-492d-97a9-33caa5d83443\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") "
Apr 17 17:18:22.654590 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.654508    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") pod \"9d96031c-66ad-492d-97a9-33caa5d83443\" (UID: \"9d96031c-66ad-492d-97a9-33caa5d83443\") "
Apr 17 17:18:22.654739 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.654712    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "9d96031c-66ad-492d-97a9-33caa5d83443" (UID: "9d96031c-66ad-492d-97a9-33caa5d83443"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:18:22.656559 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.656528    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp" (OuterVolumeSpecName: "kube-api-access-vd2dp") pod "9d96031c-66ad-492d-97a9-33caa5d83443" (UID: "9d96031c-66ad-492d-97a9-33caa5d83443"). InnerVolumeSpecName "kube-api-access-vd2dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:18:22.656673 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.656605    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d96031c-66ad-492d-97a9-33caa5d83443" (UID: "9d96031c-66ad-492d-97a9-33caa5d83443"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:18:22.680252 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.680208    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d96031c-66ad-492d-97a9-33caa5d83443" (UID: "9d96031c-66ad-492d-97a9-33caa5d83443"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:18:22.755168 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.755135    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d96031c-66ad-492d-97a9-33caa5d83443-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:18:22.755168 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.755165    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d96031c-66ad-492d-97a9-33caa5d83443-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:18:22.755168 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.755175    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd2dp\" (UniqueName: \"kubernetes.io/projected/9d96031c-66ad-492d-97a9-33caa5d83443-kube-api-access-vd2dp\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:18:22.755396 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:22.755189    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d96031c-66ad-492d-97a9-33caa5d83443-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:18:23.137355 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.137319    2579 generic.go:358] "Generic (PLEG): container finished" podID="9d96031c-66ad-492d-97a9-33caa5d83443" containerID="e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55" exitCode=0
Apr 17 17:18:23.137560 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.137406    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"
Apr 17 17:18:23.137560 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.137407    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerDied","Data":"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"}
Apr 17 17:18:23.137560 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.137456    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28" event={"ID":"9d96031c-66ad-492d-97a9-33caa5d83443","Type":"ContainerDied","Data":"938fb7bdbe91be47e09b43b2d1c3af41c2c7c8d5c2195956610ccc644bb649e2"}
Apr 17 17:18:23.137560 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.137478    2579 scope.go:117] "RemoveContainer" containerID="f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"
Apr 17 17:18:23.146890 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.146866    2579 scope.go:117] "RemoveContainer" containerID="e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"
Apr 17 17:18:23.154701 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.154681    2579 scope.go:117] "RemoveContainer" containerID="d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"
Apr 17 17:18:23.162564 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.162543    2579 scope.go:117] "RemoveContainer" containerID="f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"
Apr 17 17:18:23.162911 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:18:23.162882    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5\": container with ID starting with f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5 not found: ID does not exist" containerID="f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"
Apr 17 17:18:23.162990 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.162921    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5"} err="failed to get container status \"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5\": rpc error: code = NotFound desc = could not find container \"f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5\": container with ID starting with f8982c1943fc2c191656ff36387852fdf40147d02431db4a4d105fce310f09e5 not found: ID does not exist"
Apr 17 17:18:23.162990 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.162943    2579 scope.go:117] "RemoveContainer" containerID="e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"
Apr 17 17:18:23.163218 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:18:23.163203    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55\": container with ID starting with e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55 not found: ID does not exist" containerID="e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"
Apr 17 17:18:23.163280 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.163222    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55"} err="failed to get container status \"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55\": rpc error: code = NotFound desc = could not find container \"e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55\": container with ID starting with e5c0d469543c9a81d7a5d5415c11dc20e8af6144d70a97388b6e3f45e8b2ca55 not found: ID does not exist"
Apr 17 17:18:23.163280 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.163234    2579 scope.go:117] "RemoveContainer" containerID="d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"
Apr 17 17:18:23.163499 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:18:23.163468    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821\": container with ID starting with d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821 not found: ID does not exist" containerID="d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"
Apr 17 17:18:23.163499 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.163481    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:18:23.163600 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.163500    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821"} err="failed to get container status \"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821\": rpc error: code = NotFound desc = could not find container \"d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821\": container with ID starting with d61af737f7b870d5ead1f6a7a94e9ebba6e74252fb17d5b256632878369f5821 not found: ID does not exist"
Apr 17 17:18:23.167796 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.167771    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-pgv28"]
Apr 17 17:18:23.389498 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:23.389423    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" path="/var/lib/kubelet/pods/9d96031c-66ad-492d-97a9-33caa5d83443/volumes"
Apr 17 17:18:26.155844 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:26.155812    2579 generic.go:358] "Generic (PLEG): container finished" podID="5d51f672-715a-49c1-aaf6-b7608482442c" containerID="793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227" exitCode=0
Apr 17 17:18:26.156242 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:26.155877    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerDied","Data":"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"}
Apr 17 17:18:27.161005 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:27.160970    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerStarted","Data":"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"}
Apr 17 17:18:27.161005 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:27.161007    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerStarted","Data":"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"}
Apr 17 17:18:27.161481 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:27.161242    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:27.161481 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:27.161278    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:18:27.180813 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:27.180732    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podStartSLOduration=6.180710127 podStartE2EDuration="6.180710127s" podCreationTimestamp="2026-04-17 17:18:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:18:27.178472154 +0000 UTC m=+2834.419130589" watchObservedRunningTime="2026-04-17 17:18:27.180710127 +0000 UTC m=+2834.421368564"
Apr 17 17:18:33.169718 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:18:33.169689    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:19:03.215959 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:03.215910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 17 17:19:13.172978 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:13.172944    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:19:21.676914 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.676879    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:19:21.677447 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.677198    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container" containerID="cri-o://6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45" gracePeriod=30
Apr 17 17:19:21.677447 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.677311    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy" containerID="cri-o://c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f" gracePeriod=30
Apr 17 17:19:21.770585 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.770544    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:19:21.771115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kube-rbac-proxy"
Apr 17 17:19:21.771115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771110    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kube-rbac-proxy"
Apr 17 17:19:21.771286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="storage-initializer"
Apr 17 17:19:21.771286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771174    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="storage-initializer"
Apr 17 17:19:21.771286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771201    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container"
Apr 17 17:19:21.771286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771232    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container"
Apr 17 17:19:21.771421 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771328    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kserve-container"
Apr 17 17:19:21.771421 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.771343    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d96031c-66ad-492d-97a9-33caa5d83443" containerName="kube-rbac-proxy"
Apr 17 17:19:21.775130 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.775110    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.777868 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.777845    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Apr 17 17:19:21.778893 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.778859    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Apr 17 17:19:21.791070 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.791046    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:19:21.858199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.858165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.858364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.858214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxb8k\" (UniqueName: \"kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.858364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.858274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.858364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.858318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.959833 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.959714    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.959833 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.959790    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vxb8k\" (UniqueName: \"kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.959833 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.959814    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.960080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.959852    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.960373 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.960347    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.960578 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.960561    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.962320 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.962300    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:21.969176 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:21.969143    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxb8k\" (UniqueName: \"kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k\") pod \"isvc-sklearn-v2-predictor-69755fbb9-94sg8\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:22.086734 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.086692    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:22.222961 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.222934    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:19:22.225313 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:19:22.225282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea4ab06_3155_4464_869a_13c744517396.slice/crio-47a97023daffac9035de3bf7ec69feb5d24d0a44b8b62d8fd1988de7c70a8d2d WatchSource:0}: Error finding container 47a97023daffac9035de3bf7ec69feb5d24d0a44b8b62d8fd1988de7c70a8d2d: Status 404 returned error can't find the container with id 47a97023daffac9035de3bf7ec69feb5d24d0a44b8b62d8fd1988de7c70a8d2d
Apr 17 17:19:22.360867 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.360828    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerStarted","Data":"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"}
Apr 17 17:19:22.361070 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.360873    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerStarted","Data":"47a97023daffac9035de3bf7ec69feb5d24d0a44b8b62d8fd1988de7c70a8d2d"}
Apr 17 17:19:22.362994 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.362970    2579 generic.go:358] "Generic (PLEG): container finished" podID="5d51f672-715a-49c1-aaf6-b7608482442c" containerID="c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f" exitCode=2
Apr 17 17:19:22.363111 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:22.363040    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerDied","Data":"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"}
Apr 17 17:19:23.165661 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:23.165620    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Apr 17 17:19:24.211958 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:24.211915    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.50:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 17 17:19:26.377618 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:26.377581    2579 generic.go:358] "Generic (PLEG): container finished" podID="8ea4ab06-3155-4464-869a-13c744517396" containerID="49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b" exitCode=0
Apr 17 17:19:26.378176 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:26.377662    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerDied","Data":"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"}
Apr 17 17:19:27.383374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.383336    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerStarted","Data":"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"}
Apr 17 17:19:27.383374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.383373    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerStarted","Data":"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"}
Apr 17 17:19:27.383867 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.383701    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:27.383867 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.383791    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:27.385329 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.385302    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:19:27.406273 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:27.406221    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podStartSLOduration=6.406203612 podStartE2EDuration="6.406203612s" podCreationTimestamp="2026-04-17 17:19:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:19:27.404727922 +0000 UTC m=+2894.645386358" watchObservedRunningTime="2026-04-17 17:19:27.406203612 +0000 UTC m=+2894.646862041"
Apr 17 17:19:28.165496 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:28.165451    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Apr 17 17:19:28.389947 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:28.389902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:19:32.937552 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:32.937527    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:19:33.061370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061262    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xvd7\" (UniqueName: \"kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7\") pod \"5d51f672-715a-49c1-aaf6-b7608482442c\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") "
Apr 17 17:19:33.061370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061336    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"5d51f672-715a-49c1-aaf6-b7608482442c\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") "
Apr 17 17:19:33.061370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061377    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location\") pod \"5d51f672-715a-49c1-aaf6-b7608482442c\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") "
Apr 17 17:19:33.061686 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061410    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls\") pod \"5d51f672-715a-49c1-aaf6-b7608482442c\" (UID: \"5d51f672-715a-49c1-aaf6-b7608482442c\") "
Apr 17 17:19:33.061748 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061697    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "5d51f672-715a-49c1-aaf6-b7608482442c" (UID: "5d51f672-715a-49c1-aaf6-b7608482442c"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:19:33.061748 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.061723    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d51f672-715a-49c1-aaf6-b7608482442c" (UID: "5d51f672-715a-49c1-aaf6-b7608482442c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:19:33.063430 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.063393    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d51f672-715a-49c1-aaf6-b7608482442c" (UID: "5d51f672-715a-49c1-aaf6-b7608482442c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:19:33.063430 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.063411    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7" (OuterVolumeSpecName: "kube-api-access-9xvd7") pod "5d51f672-715a-49c1-aaf6-b7608482442c" (UID: "5d51f672-715a-49c1-aaf6-b7608482442c"). InnerVolumeSpecName "kube-api-access-9xvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:19:33.162436 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.162392    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d51f672-715a-49c1-aaf6-b7608482442c-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:19:33.162436 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.162426    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d51f672-715a-49c1-aaf6-b7608482442c-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:19:33.162436 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.162437    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvd7\" (UniqueName: \"kubernetes.io/projected/5d51f672-715a-49c1-aaf6-b7608482442c-kube-api-access-9xvd7\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:19:33.162681 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.162448    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d51f672-715a-49c1-aaf6-b7608482442c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:19:33.394045 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.394017    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:19:33.394556 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.394530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:19:33.406074 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.406044    2579 generic.go:358] "Generic (PLEG): container finished" podID="5d51f672-715a-49c1-aaf6-b7608482442c" containerID="6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45" exitCode=0
Apr 17 17:19:33.406196 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.406109    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerDied","Data":"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"}
Apr 17 17:19:33.406196 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.406121    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"
Apr 17 17:19:33.406196 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.406141    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69" event={"ID":"5d51f672-715a-49c1-aaf6-b7608482442c","Type":"ContainerDied","Data":"61feccf5318b47f522c98fb5b6594a44da214bcd86efa43495378d2005a5c2a8"}
Apr 17 17:19:33.406196 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.406161    2579 scope.go:117] "RemoveContainer" containerID="c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"
Apr 17 17:19:33.414417 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.414401    2579 scope.go:117] "RemoveContainer" containerID="6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"
Apr 17 17:19:33.422748 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.422727    2579 scope.go:117] "RemoveContainer" containerID="793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"
Apr 17 17:19:33.430080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430060    2579 scope.go:117] "RemoveContainer" containerID="c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"
Apr 17 17:19:33.430338 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:19:33.430319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f\": container with ID starting with c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f not found: ID does not exist" containerID="c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"
Apr 17 17:19:33.430386 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430347    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f"} err="failed to get container status \"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f\": rpc error: code = NotFound desc = could not find container \"c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f\": container with ID starting with c6ed11a0d7f828f84258fe45edfe9e8807de579ede81c2791a636d02ec32139f not found: ID does not exist"
Apr 17 17:19:33.430386 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430365    2579 scope.go:117] "RemoveContainer" containerID="6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"
Apr 17 17:19:33.430583 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:19:33.430562    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45\": container with ID starting with 6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45 not found: ID does not exist" containerID="6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"
Apr 17 17:19:33.430655 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430594    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45"} err="failed to get container status \"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45\": rpc error: code = NotFound desc = could not find container \"6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45\": container with ID starting with 6ea138d05ee12b79e3f1d3c8da9e876040b9aac0ab5cf3104cb78c608a0c1f45 not found: ID does not exist"
Apr 17 17:19:33.430655 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430617    2579 scope.go:117] "RemoveContainer" containerID="793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"
Apr 17 17:19:33.430865 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:19:33.430846    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227\": container with ID starting with 793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227 not found: ID does not exist" containerID="793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"
Apr 17 17:19:33.430929 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.430872    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227"} err="failed to get container status \"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227\": rpc error: code = NotFound desc = could not find container \"793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227\": container with ID starting with 793a5f2ca07016d33a7c0ef471d7fdc98d07fed5da47343ed00dd96e973aa227 not found: ID does not exist"
Apr 17 17:19:33.435378 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.435354    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:19:33.443981 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:33.443958    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2zh69"]
Apr 17 17:19:35.390040 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:35.389996    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" path="/var/lib/kubelet/pods/5d51f672-715a-49c1-aaf6-b7608482442c/volumes"
Apr 17 17:19:43.394682 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:43.394643    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:19:53.395140 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:19:53.395102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:20:03.395023 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:03.394983    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:20:13.394680 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:13.394644    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:20:23.395430 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:23.395384    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:20:33.395790 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:33.395701    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:20:41.967870 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:41.967838    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:20:41.968345 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:41.968256    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" containerID="cri-o://154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4" gracePeriod=30
Apr 17 17:20:41.968345 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:41.968285    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kube-rbac-proxy" containerID="cri-o://14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00" gracePeriod=30
Apr 17 17:20:42.044814 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.044773    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:20:42.045200 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container"
Apr 17 17:20:42.045295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045203    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container"
Apr 17 17:20:42.045295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy"
Apr 17 17:20:42.045295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045244    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy"
Apr 17 17:20:42.045295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045260    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="storage-initializer"
Apr 17 17:20:42.045295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045270    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="storage-initializer"
Apr 17 17:20:42.045550 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045361    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kube-rbac-proxy"
Apr 17 17:20:42.045550 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.045376    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d51f672-715a-49c1-aaf6-b7608482442c" containerName="kserve-container"
Apr 17 17:20:42.048804 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.048781    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.051054 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.051032    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Apr 17 17:20:42.051611 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.051592    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Apr 17 17:20:42.062146 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.062125    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:20:42.159967 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.159925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.160157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.159974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.160157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.160026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gsq9\" (UniqueName: \"kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.160157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.160102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261174 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261075    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261174 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261136    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261174 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261171    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4gsq9\" (UniqueName: \"kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261489 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261223    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261489 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:20:42.261303    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Apr 17 17:20:42.261489 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:20:42.261373    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls podName:bd22dd1f-b903-46d0-a54a-e29bb0d0014f nodeName:}" failed. No retries permitted until 2026-04-17 17:20:42.761355903 +0000 UTC m=+2970.002014456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" (UID: "bd22dd1f-b903-46d0-a54a-e29bb0d0014f") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Apr 17 17:20:42.261662 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.261836 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.261819    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.270524 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.270502    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gsq9\" (UniqueName: \"kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.642900 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.642866    2579 generic.go:358] "Generic (PLEG): container finished" podID="8ea4ab06-3155-4464-869a-13c744517396" containerID="14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00" exitCode=2
Apr 17 17:20:42.643079 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.642929    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerDied","Data":"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"}
Apr 17 17:20:42.766431 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.766381    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.768815 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.768786    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:42.959653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:42.959561    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:43.100014 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.099975    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:20:43.102958 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:20:43.102928    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd22dd1f_b903_46d0_a54a_e29bb0d0014f.slice/crio-45008835d7e3ec7d340ab2dabe2bad254c19a28a1243dd5bdefaf3d0b2d3cfdb WatchSource:0}: Error finding container 45008835d7e3ec7d340ab2dabe2bad254c19a28a1243dd5bdefaf3d0b2d3cfdb: Status 404 returned error can't find the container with id 45008835d7e3ec7d340ab2dabe2bad254c19a28a1243dd5bdefaf3d0b2d3cfdb
Apr 17 17:20:43.104993 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.104978    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:20:43.390449 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.390408    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Apr 17 17:20:43.395477 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.395455    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 17 17:20:43.648408 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.648312    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerStarted","Data":"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"}
Apr 17 17:20:43.648408 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:43.648354    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerStarted","Data":"45008835d7e3ec7d340ab2dabe2bad254c19a28a1243dd5bdefaf3d0b2d3cfdb"}
Apr 17 17:20:46.517776 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.517739    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:20:46.600970 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.600875    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls\") pod \"8ea4ab06-3155-4464-869a-13c744517396\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") "
Apr 17 17:20:46.600970 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.600932    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location\") pod \"8ea4ab06-3155-4464-869a-13c744517396\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") "
Apr 17 17:20:46.601202 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.600994    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"8ea4ab06-3155-4464-869a-13c744517396\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") "
Apr 17 17:20:46.601202 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.601038    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxb8k\" (UniqueName: \"kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k\") pod \"8ea4ab06-3155-4464-869a-13c744517396\" (UID: \"8ea4ab06-3155-4464-869a-13c744517396\") "
Apr 17 17:20:46.601305 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.601264    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ea4ab06-3155-4464-869a-13c744517396" (UID: "8ea4ab06-3155-4464-869a-13c744517396"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:20:46.601350 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.601299    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "8ea4ab06-3155-4464-869a-13c744517396" (UID: "8ea4ab06-3155-4464-869a-13c744517396"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:20:46.603072 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.603047    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ea4ab06-3155-4464-869a-13c744517396" (UID: "8ea4ab06-3155-4464-869a-13c744517396"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:20:46.603154 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.603107    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k" (OuterVolumeSpecName: "kube-api-access-vxb8k") pod "8ea4ab06-3155-4464-869a-13c744517396" (UID: "8ea4ab06-3155-4464-869a-13c744517396"). InnerVolumeSpecName "kube-api-access-vxb8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:20:46.661132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.661095    2579 generic.go:358] "Generic (PLEG): container finished" podID="8ea4ab06-3155-4464-869a-13c744517396" containerID="154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4" exitCode=0
Apr 17 17:20:46.661323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.661175    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"
Apr 17 17:20:46.661323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.661175    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerDied","Data":"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"}
Apr 17 17:20:46.661323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.661219    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8" event={"ID":"8ea4ab06-3155-4464-869a-13c744517396","Type":"ContainerDied","Data":"47a97023daffac9035de3bf7ec69feb5d24d0a44b8b62d8fd1988de7c70a8d2d"}
Apr 17 17:20:46.661323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.661244    2579 scope.go:117] "RemoveContainer" containerID="14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"
Apr 17 17:20:46.669643 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.669533    2579 scope.go:117] "RemoveContainer" containerID="154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"
Apr 17 17:20:46.676838 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.676821    2579 scope.go:117] "RemoveContainer" containerID="49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"
Apr 17 17:20:46.683770 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.683743    2579 scope.go:117] "RemoveContainer" containerID="14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"
Apr 17 17:20:46.683982 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:20:46.683965    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00\": container with ID starting with 14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00 not found: ID does not exist" containerID="14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"
Apr 17 17:20:46.684021 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.683989    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00"} err="failed to get container status \"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00\": rpc error: code = NotFound desc = could not find container \"14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00\": container with ID starting with 14f215ff0da2732b51157d39a7f4f298d36b3512f68c5d703c8a6cd1efeb7c00 not found: ID does not exist"
Apr 17 17:20:46.684021 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.684006    2579 scope.go:117] "RemoveContainer" containerID="154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"
Apr 17 17:20:46.684194 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:20:46.684180    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4\": container with ID starting with 154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4 not found: ID does not exist" containerID="154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"
Apr 17 17:20:46.684228 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.684198    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4"} err="failed to get container status \"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4\": rpc error: code = NotFound desc = could not find container \"154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4\": container with ID starting with 154d93d74579ce360403bcbf8f93940c4320d84d501436c4f1263c1424f3b8b4 not found: ID does not exist"
Apr 17 17:20:46.684228 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.684210    2579 scope.go:117] "RemoveContainer" containerID="49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"
Apr 17 17:20:46.684421 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:20:46.684404    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b\": container with ID starting with 49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b not found: ID does not exist" containerID="49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"
Apr 17 17:20:46.684457 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.684429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b"} err="failed to get container status \"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b\": rpc error: code = NotFound desc = could not find container \"49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b\": container with ID starting with 49d23d390fbd05e38937e82cf6b89c0489738fa38c201d2b45e575f524f39c7b not found: ID does not exist"
Apr 17 17:20:46.688991 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.688972    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:20:46.700268 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.700244    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-94sg8"]
Apr 17 17:20:46.702077 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.702063    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea4ab06-3155-4464-869a-13c744517396-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:20:46.702132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.702080    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea4ab06-3155-4464-869a-13c744517396-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:20:46.702132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.702091    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea4ab06-3155-4464-869a-13c744517396-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:20:46.702132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:46.702101    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxb8k\" (UniqueName: \"kubernetes.io/projected/8ea4ab06-3155-4464-869a-13c744517396-kube-api-access-vxb8k\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:20:47.392189 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:47.392152    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea4ab06-3155-4464-869a-13c744517396" path="/var/lib/kubelet/pods/8ea4ab06-3155-4464-869a-13c744517396/volumes"
Apr 17 17:20:47.665828 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:47.665799    2579 generic.go:358] "Generic (PLEG): container finished" podID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerID="64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053" exitCode=0
Apr 17 17:20:47.666281 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:47.665876    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerDied","Data":"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"}
Apr 17 17:20:48.672246 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:48.672214    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerStarted","Data":"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"}
Apr 17 17:20:48.672629 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:48.672256    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerStarted","Data":"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"}
Apr 17 17:20:48.672629 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:48.672498    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:48.693520 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:48.693471    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podStartSLOduration=6.693458618 podStartE2EDuration="6.693458618s" podCreationTimestamp="2026-04-17 17:20:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:20:48.691229248 +0000 UTC m=+2975.931887684" watchObservedRunningTime="2026-04-17 17:20:48.693458618 +0000 UTC m=+2975.934117054"
Apr 17 17:20:49.676284 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:49.676252    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:49.677884 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:49.677855    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:20:50.679210 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:50.679168    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:20:55.683604 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:55.683574    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:20:55.684208 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:20:55.684181    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:05.684633 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:05.684596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:13.518164 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:13.518133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:21:13.522128 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:13.522105    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:21:13.525424 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:13.525401    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:21:13.528874 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:13.528853    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:21:15.684860 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:15.684824    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:25.684550 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:25.684511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:35.684415 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:35.684367    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:45.684165 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:45.684118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:21:55.685107 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:21:55.685079    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:22:02.120735 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:02.120701    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:22:02.121435 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:02.121033    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" containerID="cri-o://ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9" gracePeriod=30
Apr 17 17:22:02.121435 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:02.121082    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kube-rbac-proxy" containerID="cri-o://db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4" gracePeriod=30
Apr 17 17:22:02.925701 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:02.925662    2579 generic.go:358] "Generic (PLEG): container finished" podID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerID="db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4" exitCode=2
Apr 17 17:22:02.925904 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:02.925723    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerDied","Data":"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"}
Apr 17 17:22:05.679645 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:05.679599    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 17 17:22:05.684992 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:05.684962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 17 17:22:06.672723 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.672693    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:22:06.761164 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761120    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gsq9\" (UniqueName: \"kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9\") pod \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") "
Apr 17 17:22:06.761653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761220    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") "
Apr 17 17:22:06.761653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761247    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location\") pod \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") "
Apr 17 17:22:06.761653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761275    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") pod \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\" (UID: \"bd22dd1f-b903-46d0-a54a-e29bb0d0014f\") "
Apr 17 17:22:06.761653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761562    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd22dd1f-b903-46d0-a54a-e29bb0d0014f" (UID: "bd22dd1f-b903-46d0-a54a-e29bb0d0014f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:22:06.761653 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.761590    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "bd22dd1f-b903-46d0-a54a-e29bb0d0014f" (UID: "bd22dd1f-b903-46d0-a54a-e29bb0d0014f"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:22:06.763376 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.763356    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9" (OuterVolumeSpecName: "kube-api-access-4gsq9") pod "bd22dd1f-b903-46d0-a54a-e29bb0d0014f" (UID: "bd22dd1f-b903-46d0-a54a-e29bb0d0014f"). InnerVolumeSpecName "kube-api-access-4gsq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:22:06.763539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.763521    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd22dd1f-b903-46d0-a54a-e29bb0d0014f" (UID: "bd22dd1f-b903-46d0-a54a-e29bb0d0014f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:22:06.862255 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.862222    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:22:06.862255 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.862251    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:22:06.862255 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.862262    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:22:06.862487 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.862271    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gsq9\" (UniqueName: \"kubernetes.io/projected/bd22dd1f-b903-46d0-a54a-e29bb0d0014f-kube-api-access-4gsq9\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:22:06.945317 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.945279    2579 generic.go:358] "Generic (PLEG): container finished" podID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerID="ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9" exitCode=0
Apr 17 17:22:06.945317 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.945320    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerDied","Data":"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"}
Apr 17 17:22:06.945576 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.945343    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc" event={"ID":"bd22dd1f-b903-46d0-a54a-e29bb0d0014f","Type":"ContainerDied","Data":"45008835d7e3ec7d340ab2dabe2bad254c19a28a1243dd5bdefaf3d0b2d3cfdb"}
Apr 17 17:22:06.945576 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.945357    2579 scope.go:117] "RemoveContainer" containerID="db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"
Apr 17 17:22:06.945576 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.945385    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"
Apr 17 17:22:06.953627 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.953606    2579 scope.go:117] "RemoveContainer" containerID="ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"
Apr 17 17:22:06.960497 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.960479    2579 scope.go:117] "RemoveContainer" containerID="64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"
Apr 17 17:22:06.966901 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.966877    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:22:06.967885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.967738    2579 scope.go:117] "RemoveContainer" containerID="db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"
Apr 17 17:22:06.968138 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:22:06.968095    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4\": container with ID starting with db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4 not found: ID does not exist" containerID="db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"
Apr 17 17:22:06.968194 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.968162    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4"} err="failed to get container status \"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4\": rpc error: code = NotFound desc = could not find container \"db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4\": container with ID starting with db2c3032f18f97fa65ddbbd5ace10e474f68e05045a0a5066c4b6debeb298dc4 not found: ID does not exist"
Apr 17 17:22:06.968242 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.968203    2579 scope.go:117] "RemoveContainer" containerID="ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"
Apr 17 17:22:06.968470 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:22:06.968451    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9\": container with ID starting with ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9 not found: ID does not exist" containerID="ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"
Apr 17 17:22:06.968577 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.968478    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9"} err="failed to get container status \"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9\": rpc error: code = NotFound desc = could not find container \"ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9\": container with ID starting with ada87dc571bc95f669b604e1364b81c09f0cac73f335d016c7981fff0cfcc1b9 not found: ID does not exist"
Apr 17 17:22:06.968577 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.968495    2579 scope.go:117] "RemoveContainer" containerID="64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"
Apr 17 17:22:06.968812 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:22:06.968787    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053\": container with ID starting with 64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053 not found: ID does not exist" containerID="64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"
Apr 17 17:22:06.968972 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.968818    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053"} err="failed to get container status \"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053\": rpc error: code = NotFound desc = could not find container \"64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053\": container with ID starting with 64270b48df2e9c99df7f70b5e5ea6df746edb4f26b34f497ef589bb56a6d2053 not found: ID does not exist"
Apr 17 17:22:06.970878 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:06.970858    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-h44hc"]
Apr 17 17:22:07.389860 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:22:07.389823    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" path="/var/lib/kubelet/pods/bd22dd1f-b903-46d0-a54a-e29bb0d0014f/volumes"
Apr 17 17:23:24.784039 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784006    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="storage-initializer"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784377    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="storage-initializer"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784390    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kube-rbac-proxy"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784400    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kube-rbac-proxy"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784412    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="storage-initializer"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784418    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="storage-initializer"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784428    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kube-rbac-proxy"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784433    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kube-rbac-proxy"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784441    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784448    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784458    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container"
Apr 17 17:23:24.784511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784463    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container"
Apr 17 17:23:24.785022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784521    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kserve-container"
Apr 17 17:23:24.785022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784529    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kserve-container"
Apr 17 17:23:24.785022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784537    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="8ea4ab06-3155-4464-869a-13c744517396" containerName="kube-rbac-proxy"
Apr 17 17:23:24.785022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.784543    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd22dd1f-b903-46d0-a54a-e29bb0d0014f" containerName="kube-rbac-proxy"
Apr 17 17:23:24.787798 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.787777    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:24.789967 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.789942    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Apr 17 17:23:24.789967 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.789961    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:23:24.790121 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.790015    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Apr 17 17:23:24.790574 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.790556    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:23:24.790678 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.790588    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:23:24.797561 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.797537    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:23:24.902957 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.902924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:24.903126 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.902979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:24.903126 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.903032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:24.903126 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:24.903055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xjq\" (UniqueName: \"kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.003892 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.003859    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.004081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.003909    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.004081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.003955    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.004081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.003978    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-59xjq\" (UniqueName: \"kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.004369 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.004348    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.004705 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.004681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.006449 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.006423    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.012418 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.012395    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xjq\" (UniqueName: \"kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq\") pod \"isvc-triton-predictor-84bb65d94b-2fxfg\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.101374 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.101300    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:23:25.223797 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:25.223772    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:23:25.225807 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:23:25.225778    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda319cacb_e32a_48ed_89fb_427635cdbecd.slice/crio-37a1687dafe3c15ee0efbada673c0cca76edf82fc86d1ea41cb3bf581690cd9d WatchSource:0}: Error finding container 37a1687dafe3c15ee0efbada673c0cca76edf82fc86d1ea41cb3bf581690cd9d: Status 404 returned error can't find the container with id 37a1687dafe3c15ee0efbada673c0cca76edf82fc86d1ea41cb3bf581690cd9d
Apr 17 17:23:26.217655 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:26.217611    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerStarted","Data":"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"}
Apr 17 17:23:26.217655 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:26.217647    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerStarted","Data":"37a1687dafe3c15ee0efbada673c0cca76edf82fc86d1ea41cb3bf581690cd9d"}
Apr 17 17:23:29.228941 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:29.228850    2579 generic.go:358] "Generic (PLEG): container finished" podID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerID="ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4" exitCode=0
Apr 17 17:23:29.228941 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:23:29.228899    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerDied","Data":"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"}
Apr 17 17:25:23.690815 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:23.690787    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerStarted","Data":"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"}
Apr 17 17:25:24.696786 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:24.696730    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerStarted","Data":"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"}
Apr 17 17:25:24.697288 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:24.696860    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:24.717580 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:24.717524    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podStartSLOduration=6.447038497 podStartE2EDuration="2m0.717508463s" podCreationTimestamp="2026-04-17 17:23:24 +0000 UTC" firstStartedPulling="2026-04-17 17:23:29.229959051 +0000 UTC m=+3136.470617465" lastFinishedPulling="2026-04-17 17:25:23.500429016 +0000 UTC m=+3250.741087431" observedRunningTime="2026-04-17 17:25:24.715973679 +0000 UTC m=+3251.956632119" watchObservedRunningTime="2026-04-17 17:25:24.717508463 +0000 UTC m=+3251.958166900"
Apr 17 17:25:25.700463 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:25.700430    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:25.701805 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:25.701755    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Apr 17 17:25:26.703751 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:26.703714    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Apr 17 17:25:31.708129 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:31.708100    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:31.708893 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:31.708872    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:36.350789 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.350719    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:25:36.351216 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.351180    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container" containerID="cri-o://889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a" gracePeriod=30
Apr 17 17:25:36.351286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.351213    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kube-rbac-proxy" containerID="cri-o://5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb" gracePeriod=30
Apr 17 17:25:36.703946 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.703904    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 17 17:25:36.738364 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.738325    2579 generic.go:358] "Generic (PLEG): container finished" podID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerID="5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb" exitCode=2
Apr 17 17:25:36.738543 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:36.738392    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerDied","Data":"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"}
Apr 17 17:25:39.496227 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.496201    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:39.596753 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.596713    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config\") pod \"a319cacb-e32a-48ed-89fb-427635cdbecd\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") "
Apr 17 17:25:39.596952 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.596835    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59xjq\" (UniqueName: \"kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq\") pod \"a319cacb-e32a-48ed-89fb-427635cdbecd\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") "
Apr 17 17:25:39.596952 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.596868    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls\") pod \"a319cacb-e32a-48ed-89fb-427635cdbecd\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") "
Apr 17 17:25:39.597055 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.596989    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location\") pod \"a319cacb-e32a-48ed-89fb-427635cdbecd\" (UID: \"a319cacb-e32a-48ed-89fb-427635cdbecd\") "
Apr 17 17:25:39.597317 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.597287    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "a319cacb-e32a-48ed-89fb-427635cdbecd" (UID: "a319cacb-e32a-48ed-89fb-427635cdbecd"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:25:39.598022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.597431    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a319cacb-e32a-48ed-89fb-427635cdbecd" (UID: "a319cacb-e32a-48ed-89fb-427635cdbecd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:25:39.599150 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.599122    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a319cacb-e32a-48ed-89fb-427635cdbecd" (UID: "a319cacb-e32a-48ed-89fb-427635cdbecd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:25:39.599331 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.599310    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq" (OuterVolumeSpecName: "kube-api-access-59xjq") pod "a319cacb-e32a-48ed-89fb-427635cdbecd" (UID: "a319cacb-e32a-48ed-89fb-427635cdbecd"). InnerVolumeSpecName "kube-api-access-59xjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:25:39.698710 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.698606    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a319cacb-e32a-48ed-89fb-427635cdbecd-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:25:39.698710 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.698639    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a319cacb-e32a-48ed-89fb-427635cdbecd-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:25:39.698710 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.698650    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a319cacb-e32a-48ed-89fb-427635cdbecd-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:25:39.698710 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.698662    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xjq\" (UniqueName: \"kubernetes.io/projected/a319cacb-e32a-48ed-89fb-427635cdbecd-kube-api-access-59xjq\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:25:39.750334 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.750296    2579 generic.go:358] "Generic (PLEG): container finished" podID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerID="889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a" exitCode=0
Apr 17 17:25:39.750501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.750371    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"
Apr 17 17:25:39.750501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.750374    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerDied","Data":"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"}
Apr 17 17:25:39.750501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.750482    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg" event={"ID":"a319cacb-e32a-48ed-89fb-427635cdbecd","Type":"ContainerDied","Data":"37a1687dafe3c15ee0efbada673c0cca76edf82fc86d1ea41cb3bf581690cd9d"}
Apr 17 17:25:39.750501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.750501    2579 scope.go:117] "RemoveContainer" containerID="5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"
Apr 17 17:25:39.759204 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.759185    2579 scope.go:117] "RemoveContainer" containerID="889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"
Apr 17 17:25:39.767464 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.767442    2579 scope.go:117] "RemoveContainer" containerID="ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"
Apr 17 17:25:39.774135 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.774106    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:25:39.775447 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.775415    2579 scope.go:117] "RemoveContainer" containerID="5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"
Apr 17 17:25:39.775730 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:25:39.775710    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb\": container with ID starting with 5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb not found: ID does not exist" containerID="5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"
Apr 17 17:25:39.775828 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.775741    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb"} err="failed to get container status \"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb\": rpc error: code = NotFound desc = could not find container \"5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb\": container with ID starting with 5fbefcd42a32cb4b89b831d3f4c4abc3a20727e5757d96bf378222b678d675fb not found: ID does not exist"
Apr 17 17:25:39.775828 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.775783    2579 scope.go:117] "RemoveContainer" containerID="889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"
Apr 17 17:25:39.776214 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:25:39.776173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a\": container with ID starting with 889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a not found: ID does not exist" containerID="889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"
Apr 17 17:25:39.776386 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.776253    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a"} err="failed to get container status \"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a\": rpc error: code = NotFound desc = could not find container \"889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a\": container with ID starting with 889b6035cfd86152a1a591c0d14182cb5acf1fb0566e20d9c171420c58c0004a not found: ID does not exist"
Apr 17 17:25:39.776386 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.776312    2579 scope.go:117] "RemoveContainer" containerID="ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"
Apr 17 17:25:39.776698 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:25:39.776638    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4\": container with ID starting with ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4 not found: ID does not exist" containerID="ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"
Apr 17 17:25:39.776698 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.776676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4"} err="failed to get container status \"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4\": rpc error: code = NotFound desc = could not find container \"ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4\": container with ID starting with ff7523c6b994a33f5c2f0c4a6aafdfd8563030ff5859c1b22b0cd16694ccd1b4 not found: ID does not exist"
Apr 17 17:25:39.778370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:39.778346    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2fxfg"]
Apr 17 17:25:41.389175 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:25:41.389142    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" path="/var/lib/kubelet/pods/a319cacb-e32a-48ed-89fb-427635cdbecd/volumes"
Apr 17 17:26:13.568210 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:26:13.568167    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:26:13.568809 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:26:13.568168    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:26:13.572621 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:26:13.572598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:26:13.572621 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:26:13.572610    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:27:16.773005 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.772969    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773290    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kube-rbac-proxy"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773300    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kube-rbac-proxy"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773316    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="storage-initializer"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773322    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="storage-initializer"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773338    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773389    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kube-rbac-proxy"
Apr 17 17:27:16.773479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.773402    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="a319cacb-e32a-48ed-89fb-427635cdbecd" containerName="kserve-container"
Apr 17 17:27:16.776784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.776749    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:16.780501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.780477    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:27:16.780650 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.780632    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:27:16.780964 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.780947    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 17 17:27:16.781154 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.781138    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Apr 17 17:27:16.781245 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.781141    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:27:16.787724 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.787704    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:27:16.914468 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.914439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:16.914617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.914490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6gsv\" (UniqueName: \"kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:16.914617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.914520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:16.914617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:16.914565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.015885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.015842    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.016080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.015913    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t6gsv\" (UniqueName: \"kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.016080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.015940    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.016080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.016018    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.016368 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.016347    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.016516 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.016498    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.018362 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.018346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.026997 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.026944    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6gsv\" (UniqueName: \"kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.088467 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.088437    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:17.224987 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.224903    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:27:17.227439 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:27:17.227400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad245664_7dbf_4f9c_a6fa_ff3a961ae6eb.slice/crio-a5737d3be1d2e5c26fba2dfeb83cca685dde7ea43648cc2776adffdcf5c16973 WatchSource:0}: Error finding container a5737d3be1d2e5c26fba2dfeb83cca685dde7ea43648cc2776adffdcf5c16973: Status 404 returned error can't find the container with id a5737d3be1d2e5c26fba2dfeb83cca685dde7ea43648cc2776adffdcf5c16973
Apr 17 17:27:17.229312 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:17.229297    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:27:18.069153 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:18.069123    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerStarted","Data":"49a44fda453c1f199af011d0a18c09f788002ac65e426f4247545fd6ad8754c4"}
Apr 17 17:27:18.069153 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:18.069154    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerStarted","Data":"a5737d3be1d2e5c26fba2dfeb83cca685dde7ea43648cc2776adffdcf5c16973"}
Apr 17 17:27:22.082892 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:22.082856    2579 generic.go:358] "Generic (PLEG): container finished" podID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerID="49a44fda453c1f199af011d0a18c09f788002ac65e426f4247545fd6ad8754c4" exitCode=0
Apr 17 17:27:22.083256 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:22.082926    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerDied","Data":"49a44fda453c1f199af011d0a18c09f788002ac65e426f4247545fd6ad8754c4"}
Apr 17 17:27:23.087850 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:23.087810    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerStarted","Data":"e190409b5e7cbbd13a171180d1ed02e1e4bd9ccd90efa0945f47cb4d0f9142fb"}
Apr 17 17:27:23.088261 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:23.087859    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerStarted","Data":"80fa0d653bc91bec706e6e2bb5d37d89d6ce9c95c4daa680f224459a1600bd63"}
Apr 17 17:27:23.088261 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:23.088059    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:23.107071 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:23.107021    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" podStartSLOduration=7.107008605 podStartE2EDuration="7.107008605s" podCreationTimestamp="2026-04-17 17:27:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:27:23.105981208 +0000 UTC m=+3370.346639643" watchObservedRunningTime="2026-04-17 17:27:23.107008605 +0000 UTC m=+3370.347667040"
Apr 17 17:27:24.091437 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:24.091399    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:27:30.100659 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:27:30.100631    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:28:00.104813 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:00.104781    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:28:06.786417 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.786383    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:28:06.786894 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.786831    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kserve-container" containerID="cri-o://80fa0d653bc91bec706e6e2bb5d37d89d6ce9c95c4daa680f224459a1600bd63" gracePeriod=30
Apr 17 17:28:06.786972 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.786864    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kube-rbac-proxy" containerID="cri-o://e190409b5e7cbbd13a171180d1ed02e1e4bd9ccd90efa0945f47cb4d0f9142fb" gracePeriod=30
Apr 17 17:28:06.880830 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.880798    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:28:06.884526 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.884507    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:06.886915 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.886886    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Apr 17 17:28:06.887043 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.886916    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 17 17:28:06.896473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.896449    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:28:06.954233 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.954202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbv4\" (UniqueName: \"kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:06.954393 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.954267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:06.954393 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.954310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:06.954393 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:06.954340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.054986 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.054904    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbv4\" (UniqueName: \"kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.055160 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.054998    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.055160 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.055036    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.055160 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.055076    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.055344 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:28:07.055199    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Apr 17 17:28:07.055344 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:28:07.055284    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls podName:3deafb57-e19f-4ff6-972a-4eaab3b078fb nodeName:}" failed. No retries permitted until 2026-04-17 17:28:07.5552517 +0000 UTC m=+3414.795910116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-hk67v" (UID: "3deafb57-e19f-4ff6-972a-4eaab3b078fb") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Apr 17 17:28:07.055477 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.055389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.055780 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.055741    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.064555 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.064524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbv4\" (UniqueName: \"kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.246993 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.246959    2579 generic.go:358] "Generic (PLEG): container finished" podID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerID="e190409b5e7cbbd13a171180d1ed02e1e4bd9ccd90efa0945f47cb4d0f9142fb" exitCode=2
Apr 17 17:28:07.247159 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.247035    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerDied","Data":"e190409b5e7cbbd13a171180d1ed02e1e4bd9ccd90efa0945f47cb4d0f9142fb"}
Apr 17 17:28:07.560228 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.560175    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.562582 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.562563    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-hk67v\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:07.795913 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:07.795877    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:08.195186 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:08.195161    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:28:08.197716 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:28:08.197685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deafb57_e19f_4ff6_972a_4eaab3b078fb.slice/crio-0262a0fb6d32efaea12f2c3584517b958208b273ccacec83a05d8ce289cba90c WatchSource:0}: Error finding container 0262a0fb6d32efaea12f2c3584517b958208b273ccacec83a05d8ce289cba90c: Status 404 returned error can't find the container with id 0262a0fb6d32efaea12f2c3584517b958208b273ccacec83a05d8ce289cba90c
Apr 17 17:28:08.251499 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:08.251471    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerStarted","Data":"0262a0fb6d32efaea12f2c3584517b958208b273ccacec83a05d8ce289cba90c"}
Apr 17 17:28:09.256253 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:09.256216    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerStarted","Data":"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"}
Apr 17 17:28:10.095667 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:10.095628    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Apr 17 17:28:10.101059 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:10.101031    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Apr 17 17:28:12.268470 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:12.268431    2579 generic.go:358] "Generic (PLEG): container finished" podID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerID="af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e" exitCode=0
Apr 17 17:28:12.268899 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:12.268505    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerDied","Data":"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"}
Apr 17 17:28:13.275829 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.275796    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerStarted","Data":"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"}
Apr 17 17:28:13.276255 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.275839    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerStarted","Data":"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"}
Apr 17 17:28:13.276323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.276254    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:13.276323 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.276289    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:13.278875 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.278847    2579 generic.go:358] "Generic (PLEG): container finished" podID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerID="80fa0d653bc91bec706e6e2bb5d37d89d6ce9c95c4daa680f224459a1600bd63" exitCode=0
Apr 17 17:28:13.279004 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.278909    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerDied","Data":"80fa0d653bc91bec706e6e2bb5d37d89d6ce9c95c4daa680f224459a1600bd63"}
Apr 17 17:28:13.302528 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.302477    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" podStartSLOduration=7.302463858 podStartE2EDuration="7.302463858s" podCreationTimestamp="2026-04-17 17:28:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:28:13.301547083 +0000 UTC m=+3420.542205552" watchObservedRunningTime="2026-04-17 17:28:13.302463858 +0000 UTC m=+3420.543122294"
Apr 17 17:28:13.330853 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.330834    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:28:13.410795 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.410704    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") "
Apr 17 17:28:13.410795 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.410779    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location\") pod \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") "
Apr 17 17:28:13.411014 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.410799    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls\") pod \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") "
Apr 17 17:28:13.411014 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.410899    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6gsv\" (UniqueName: \"kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv\") pod \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\" (UID: \"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb\") "
Apr 17 17:28:13.411132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.411081    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" (UID: "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:28:13.411132 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.411117    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" (UID: "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:28:13.411256 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.411212    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:28:13.411256 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.411233    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:28:13.412728 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.412707    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" (UID: "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:28:13.412876 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.412855    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv" (OuterVolumeSpecName: "kube-api-access-t6gsv") pod "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" (UID: "ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb"). InnerVolumeSpecName "kube-api-access-t6gsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:28:13.512452 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.512419    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6gsv\" (UniqueName: \"kubernetes.io/projected/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-kube-api-access-t6gsv\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:28:13.512452 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:13.512454    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:28:14.283923 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.283884    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv" event={"ID":"ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb","Type":"ContainerDied","Data":"a5737d3be1d2e5c26fba2dfeb83cca685dde7ea43648cc2776adffdcf5c16973"}
Apr 17 17:28:14.283923 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.283912    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"
Apr 17 17:28:14.284385 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.283941    2579 scope.go:117] "RemoveContainer" containerID="e190409b5e7cbbd13a171180d1ed02e1e4bd9ccd90efa0945f47cb4d0f9142fb"
Apr 17 17:28:14.293199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.293134    2579 scope.go:117] "RemoveContainer" containerID="80fa0d653bc91bec706e6e2bb5d37d89d6ce9c95c4daa680f224459a1600bd63"
Apr 17 17:28:14.302566 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.302549    2579 scope.go:117] "RemoveContainer" containerID="49a44fda453c1f199af011d0a18c09f788002ac65e426f4247545fd6ad8754c4"
Apr 17 17:28:14.310349 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.310329    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:28:14.313711 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:14.313688    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7nnfv"]
Apr 17 17:28:15.389849 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:15.389812    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" path="/var/lib/kubelet/pods/ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb/volumes"
Apr 17 17:28:19.289501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:19.289471    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:49.293500 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:49.293470    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:28:56.974246 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:56.974212    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:28:56.974690 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:56.974632    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kserve-container" containerID="cri-o://28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d" gracePeriod=30
Apr 17 17:28:56.974810 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:56.974714    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kube-rbac-proxy" containerID="cri-o://12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20" gracePeriod=30
Apr 17 17:28:57.425826 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:57.425788    2579 generic.go:358] "Generic (PLEG): container finished" podID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerID="12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20" exitCode=2
Apr 17 17:28:57.425998 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:57.425846    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerDied","Data":"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"}
Apr 17 17:28:59.284477 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:28:59.284438    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Apr 17 17:29:03.418772 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.418734    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:29:03.452958 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.452927    2579 generic.go:358] "Generic (PLEG): container finished" podID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerID="28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d" exitCode=0
Apr 17 17:29:03.453108 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.453022    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"
Apr 17 17:29:03.453153 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.453010    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerDied","Data":"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"}
Apr 17 17:29:03.453153 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.453141    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v" event={"ID":"3deafb57-e19f-4ff6-972a-4eaab3b078fb","Type":"ContainerDied","Data":"0262a0fb6d32efaea12f2c3584517b958208b273ccacec83a05d8ce289cba90c"}
Apr 17 17:29:03.453218 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.453167    2579 scope.go:117] "RemoveContainer" containerID="12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"
Apr 17 17:29:03.460925 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.460904    2579 scope.go:117] "RemoveContainer" containerID="28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"
Apr 17 17:29:03.468290 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.468269    2579 scope.go:117] "RemoveContainer" containerID="af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"
Apr 17 17:29:03.475397 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.475378    2579 scope.go:117] "RemoveContainer" containerID="12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"
Apr 17 17:29:03.475685 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:29:03.475667    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20\": container with ID starting with 12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20 not found: ID does not exist" containerID="12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"
Apr 17 17:29:03.475742 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.475694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20"} err="failed to get container status \"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20\": rpc error: code = NotFound desc = could not find container \"12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20\": container with ID starting with 12eae80cc3a7cee0e378b529cb729fa86621284932b4394fe69f9d4b90caab20 not found: ID does not exist"
Apr 17 17:29:03.475742 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.475713    2579 scope.go:117] "RemoveContainer" containerID="28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"
Apr 17 17:29:03.475967 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:29:03.475948    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d\": container with ID starting with 28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d not found: ID does not exist" containerID="28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"
Apr 17 17:29:03.476015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.475974    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d"} err="failed to get container status \"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d\": rpc error: code = NotFound desc = could not find container \"28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d\": container with ID starting with 28a0d79bce9afa22b501cf4a7e32c4cb81908e0319f6b044a674cffc898c331d not found: ID does not exist"
Apr 17 17:29:03.476015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.475991    2579 scope.go:117] "RemoveContainer" containerID="af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"
Apr 17 17:29:03.476211 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:29:03.476195    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e\": container with ID starting with af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e not found: ID does not exist" containerID="af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"
Apr 17 17:29:03.476254 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.476215    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e"} err="failed to get container status \"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e\": rpc error: code = NotFound desc = could not find container \"af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e\": container with ID starting with af71a36ff06d457911424525764a6d0fd0877a971c1febf1d455be1d5912bf3e not found: ID does not exist"
Apr 17 17:29:03.559994 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.559963    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxbv4\" (UniqueName: \"kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4\") pod \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") "
Apr 17 17:29:03.560164 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.560009    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location\") pod \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") "
Apr 17 17:29:03.560164 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.560040    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") pod \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") "
Apr 17 17:29:03.560164 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.560076    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\" (UID: \"3deafb57-e19f-4ff6-972a-4eaab3b078fb\") "
Apr 17 17:29:03.560404 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.560331    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3deafb57-e19f-4ff6-972a-4eaab3b078fb" (UID: "3deafb57-e19f-4ff6-972a-4eaab3b078fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:29:03.560511 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.560487    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "3deafb57-e19f-4ff6-972a-4eaab3b078fb" (UID: "3deafb57-e19f-4ff6-972a-4eaab3b078fb"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:29:03.562048 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.562021    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4" (OuterVolumeSpecName: "kube-api-access-jxbv4") pod "3deafb57-e19f-4ff6-972a-4eaab3b078fb" (UID: "3deafb57-e19f-4ff6-972a-4eaab3b078fb"). InnerVolumeSpecName "kube-api-access-jxbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:29:03.562146 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.562080    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3deafb57-e19f-4ff6-972a-4eaab3b078fb" (UID: "3deafb57-e19f-4ff6-972a-4eaab3b078fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:29:03.661199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.661173    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxbv4\" (UniqueName: \"kubernetes.io/projected/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kube-api-access-jxbv4\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:29:03.661199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.661196    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3deafb57-e19f-4ff6-972a-4eaab3b078fb-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:29:03.661199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.661206    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3deafb57-e19f-4ff6-972a-4eaab3b078fb-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:29:03.661415 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.661217    2579 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3deafb57-e19f-4ff6-972a-4eaab3b078fb-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:29:03.775429 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.775401    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:29:03.779469 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:03.779422    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-hk67v"]
Apr 17 17:29:05.389335 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:29:05.389301    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" path="/var/lib/kubelet/pods/3deafb57-e19f-4ff6-972a-4eaab3b078fb/volumes"
Apr 17 17:30:17.324037 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.323998    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324418    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324434    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324443    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="storage-initializer"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324449    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="storage-initializer"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324462    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324481    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kserve-container"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kserve-container"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324496    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kserve-container"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324502    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kserve-container"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324514    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="storage-initializer"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324519    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="storage-initializer"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324574    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324584    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kserve-container"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324594    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad245664-7dbf-4f9c-a6fa-ff3a961ae6eb" containerName="kube-rbac-proxy"
Apr 17 17:30:17.324631 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.324600    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="3deafb57-e19f-4ff6-972a-4eaab3b078fb" containerName="kserve-container"
Apr 17 17:30:17.327885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.327868    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.330465 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.330442    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 17 17:30:17.330596 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.330463    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Apr 17 17:30:17.330596 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.330467    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:30:17.330895 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.330880    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:30:17.331167 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.331149    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:30:17.349261 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.349236    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:30:17.371463 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.371424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4z5\" (UniqueName: \"kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.371627 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.371511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.371627 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.371577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.371627 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.371617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.472679 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.472646    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4z5\" (UniqueName: \"kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.472914 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.472714    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.472914 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.472790    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.472914 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.472811    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.473092 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:30:17.472995    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Apr 17 17:30:17.473092 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:30:17.473064    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls podName:344b5ab1-c548-4704-a1e5-b590af804754 nodeName:}" failed. No retries permitted until 2026-04-17 17:30:17.973049355 +0000 UTC m=+3545.213707788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" (UID: "344b5ab1-c548-4704-a1e5-b590af804754") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Apr 17 17:30:17.473215 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.473188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.473407 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.473388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.481457 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.481433    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4z5\" (UniqueName: \"kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.976895 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.976861    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:17.979319 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:17.979294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:18.238289 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:18.238206    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:18.363950 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:18.363925    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:30:18.366800 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:30:18.366776    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344b5ab1_c548_4704_a1e5_b590af804754.slice/crio-e0f15efb1e138b37c7eef84ffb6359657f26d44cdb82e847311a9ad00347b3ff WatchSource:0}: Error finding container e0f15efb1e138b37c7eef84ffb6359657f26d44cdb82e847311a9ad00347b3ff: Status 404 returned error can't find the container with id e0f15efb1e138b37c7eef84ffb6359657f26d44cdb82e847311a9ad00347b3ff
Apr 17 17:30:18.703723 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:18.703688    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerStarted","Data":"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"}
Apr 17 17:30:18.703723 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:18.703726    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerStarted","Data":"e0f15efb1e138b37c7eef84ffb6359657f26d44cdb82e847311a9ad00347b3ff"}
Apr 17 17:30:22.720154 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:22.720112    2579 generic.go:358] "Generic (PLEG): container finished" podID="344b5ab1-c548-4704-a1e5-b590af804754" containerID="7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1" exitCode=0
Apr 17 17:30:22.720549 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:22.720171    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerDied","Data":"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"}
Apr 17 17:30:23.725253 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:23.725219    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerStarted","Data":"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"}
Apr 17 17:30:23.725716 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:23.725261    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerStarted","Data":"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"}
Apr 17 17:30:23.725716 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:23.725517    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:23.745789 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:23.745732    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" podStartSLOduration=6.745719125 podStartE2EDuration="6.745719125s" podCreationTimestamp="2026-04-17 17:30:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:30:23.743363763 +0000 UTC m=+3550.984022197" watchObservedRunningTime="2026-04-17 17:30:23.745719125 +0000 UTC m=+3550.986377559"
Apr 17 17:30:24.729128 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:24.729099    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:30:30.737233 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:30:30.737207    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:31:00.816857 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:00.816811    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 17 17:31:10.740539 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:10.740512    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:31:13.595282 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:13.595257    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:31:13.597337 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:13.597316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:31:13.599169 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:13.599145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:31:13.601142 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:13.601122    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:31:17.326851 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:17.326817    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:31:17.327433 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:17.327110    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kserve-container" containerID="cri-o://8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f" gracePeriod=30
Apr 17 17:31:17.327433 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:17.327150    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kube-rbac-proxy" containerID="cri-o://a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5" gracePeriod=30
Apr 17 17:31:17.903361 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:17.903331    2579 generic.go:358] "Generic (PLEG): container finished" podID="344b5ab1-c548-4704-a1e5-b590af804754" containerID="a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5" exitCode=2
Apr 17 17:31:17.903551 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:17.903405    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerDied","Data":"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"}
Apr 17 17:31:20.732575 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:20.732540    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Apr 17 17:31:24.764419 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.764394    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:31:24.828044 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828014    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") pod \"344b5ab1-c548-4704-a1e5-b590af804754\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") "
Apr 17 17:31:24.828221 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828116    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd4z5\" (UniqueName: \"kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5\") pod \"344b5ab1-c548-4704-a1e5-b590af804754\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") "
Apr 17 17:31:24.828221 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828140    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"344b5ab1-c548-4704-a1e5-b590af804754\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") "
Apr 17 17:31:24.828221 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828194    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location\") pod \"344b5ab1-c548-4704-a1e5-b590af804754\" (UID: \"344b5ab1-c548-4704-a1e5-b590af804754\") "
Apr 17 17:31:24.828528 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828501    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "344b5ab1-c548-4704-a1e5-b590af804754" (UID: "344b5ab1-c548-4704-a1e5-b590af804754"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:31:24.828528 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.828513    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "344b5ab1-c548-4704-a1e5-b590af804754" (UID: "344b5ab1-c548-4704-a1e5-b590af804754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:31:24.830112 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.830088    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "344b5ab1-c548-4704-a1e5-b590af804754" (UID: "344b5ab1-c548-4704-a1e5-b590af804754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:31:24.830203 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.830126    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5" (OuterVolumeSpecName: "kube-api-access-bd4z5") pod "344b5ab1-c548-4704-a1e5-b590af804754" (UID: "344b5ab1-c548-4704-a1e5-b590af804754"). InnerVolumeSpecName "kube-api-access-bd4z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:31:24.927922 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.927834    2579 generic.go:358] "Generic (PLEG): container finished" podID="344b5ab1-c548-4704-a1e5-b590af804754" containerID="8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f" exitCode=0
Apr 17 17:31:24.927922 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.927916    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"
Apr 17 17:31:24.928157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.927917    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerDied","Data":"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"}
Apr 17 17:31:24.928157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.927958    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv" event={"ID":"344b5ab1-c548-4704-a1e5-b590af804754","Type":"ContainerDied","Data":"e0f15efb1e138b37c7eef84ffb6359657f26d44cdb82e847311a9ad00347b3ff"}
Apr 17 17:31:24.928157 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.927978    2579 scope.go:117] "RemoveContainer" containerID="a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"
Apr 17 17:31:24.928836 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.928812    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4z5\" (UniqueName: \"kubernetes.io/projected/344b5ab1-c548-4704-a1e5-b590af804754-kube-api-access-bd4z5\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:31:24.928836 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.928839    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/344b5ab1-c548-4704-a1e5-b590af804754-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:31:24.929022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.928856    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/344b5ab1-c548-4704-a1e5-b590af804754-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:31:24.929022 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.928870    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/344b5ab1-c548-4704-a1e5-b590af804754-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:31:24.936222 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.936205    2579 scope.go:117] "RemoveContainer" containerID="8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"
Apr 17 17:31:24.943871 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.943855    2579 scope.go:117] "RemoveContainer" containerID="7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"
Apr 17 17:31:24.950695 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.950677    2579 scope.go:117] "RemoveContainer" containerID="a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"
Apr 17 17:31:24.950955 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:31:24.950938    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5\": container with ID starting with a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5 not found: ID does not exist" containerID="a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"
Apr 17 17:31:24.951011 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.950965    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5"} err="failed to get container status \"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5\": rpc error: code = NotFound desc = could not find container \"a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5\": container with ID starting with a3418c5d89422bc2e9371d0aa27f79a1d31244cb1c77c488133e163b54bca2d5 not found: ID does not exist"
Apr 17 17:31:24.951011 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.950983    2579 scope.go:117] "RemoveContainer" containerID="8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"
Apr 17 17:31:24.951219 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:31:24.951203    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f\": container with ID starting with 8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f not found: ID does not exist" containerID="8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"
Apr 17 17:31:24.951270 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.951227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f"} err="failed to get container status \"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f\": rpc error: code = NotFound desc = could not find container \"8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f\": container with ID starting with 8ca8821606ea4e0eb3f3aae91e1f704fa1f107e437edf17db61dcd76ccebed0f not found: ID does not exist"
Apr 17 17:31:24.951270 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.951242    2579 scope.go:117] "RemoveContainer" containerID="7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"
Apr 17 17:31:24.951440 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:31:24.951426    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1\": container with ID starting with 7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1 not found: ID does not exist" containerID="7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"
Apr 17 17:31:24.951496 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.951443    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1"} err="failed to get container status \"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1\": rpc error: code = NotFound desc = could not find container \"7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1\": container with ID starting with 7cfcedebc4108549617bf7fdc5a41e43672c1f423a02710d20a887bbecfc9bd1 not found: ID does not exist"
Apr 17 17:31:24.976187 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.976156    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:31:24.991997 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:24.991969    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-q7hxv"]
Apr 17 17:31:25.389593 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:31:25.389563    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344b5ab1-c548-4704-a1e5-b590af804754" path="/var/lib/kubelet/pods/344b5ab1-c548-4704-a1e5-b590af804754/volumes"
Apr 17 17:32:37.630528 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630430    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630935    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kserve-container"
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630951    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kserve-container"
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630969    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="storage-initializer"
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630978    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="storage-initializer"
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630988    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kube-rbac-proxy"
Apr 17 17:32:37.631066 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.630997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kube-rbac-proxy"
Apr 17 17:32:37.631272 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.631076    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kube-rbac-proxy"
Apr 17 17:32:37.631272 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.631094    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="344b5ab1-c548-4704-a1e5-b590af804754" containerName="kserve-container"
Apr 17 17:32:37.634637 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.634615    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.637248 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.637227    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Apr 17 17:32:37.637352 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.637257    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:32:37.637541 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.637517    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:32:37.638010 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.637992    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Apr 17 17:32:37.638111 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.638027    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Apr 17 17:32:37.638111 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.638033    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:32:37.649823 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.649754    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:32:37.770637 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.770609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.770830 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.770670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.770830 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.770815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmx5t\" (UniqueName: \"kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.771043 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.770854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872121 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872083    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872313 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872140    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872313 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872183    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dmx5t\" (UniqueName: \"kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872313 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872205    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872596 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872575    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.872848 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.872827    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.874605 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.874587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.881424 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.881362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmx5t\" (UniqueName: \"kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t\") pod \"isvc-sklearn-s3-predictor-88457d696-jcz4m\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:37.945714 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:37.945674    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:38.072782 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:38.072721    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:32:38.081105 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:32:38.081070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b3e9e3_76c5_4323_9c52_f90f47af846e.slice/crio-7d4000ef55b2426aeb884ee8462c62df7956ebfb320eb492694eb4d8aa3afe50 WatchSource:0}: Error finding container 7d4000ef55b2426aeb884ee8462c62df7956ebfb320eb492694eb4d8aa3afe50: Status 404 returned error can't find the container with id 7d4000ef55b2426aeb884ee8462c62df7956ebfb320eb492694eb4d8aa3afe50
Apr 17 17:32:38.082848 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:38.082828    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:32:38.175068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:38.175035    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerStarted","Data":"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"}
Apr 17 17:32:38.175068 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:38.175074    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerStarted","Data":"7d4000ef55b2426aeb884ee8462c62df7956ebfb320eb492694eb4d8aa3afe50"}
Apr 17 17:32:39.179532 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:39.179491    2579 generic.go:358] "Generic (PLEG): container finished" podID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerID="b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a" exitCode=0
Apr 17 17:32:39.179951 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:39.179579    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerDied","Data":"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"}
Apr 17 17:32:40.185113 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.185079    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerStarted","Data":"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"}
Apr 17 17:32:40.185113 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.185116    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerStarted","Data":"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"}
Apr 17 17:32:40.185559 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.185300    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:40.185559 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.185404    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:40.186714 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.186689    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:32:40.203596 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:40.203554    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podStartSLOduration=3.203539261 podStartE2EDuration="3.203539261s" podCreationTimestamp="2026-04-17 17:32:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:32:40.20212819 +0000 UTC m=+3687.442786638" watchObservedRunningTime="2026-04-17 17:32:40.203539261 +0000 UTC m=+3687.444197698"
Apr 17 17:32:41.189486 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:41.189451    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:32:46.194038 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:46.194009    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:32:46.194625 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:46.194599    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:32:56.194911 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:32:56.194875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:33:06.194478 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:06.194437    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:33:16.195091 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:16.195050    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:33:26.195349 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:26.195309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:33:36.194778 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:36.194734    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Apr 17 17:33:46.195976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:46.195946    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:33:47.726142 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.726110    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:33:47.726630 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.726369    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container" containerID="cri-o://466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6" gracePeriod=30
Apr 17 17:33:47.726630 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.726404    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kube-rbac-proxy" containerID="cri-o://368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a" gracePeriod=30
Apr 17 17:33:47.883354 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.883316    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:33:47.886991 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.886961    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:47.890321 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.890285    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Apr 17 17:33:47.890471 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.890338    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Apr 17 17:33:47.890542 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.890527    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 17 17:33:47.899687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.899660    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:33:47.973670 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.973626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:47.973877 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.973747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:47.973877 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.973814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:47.974002 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.973893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrnc\" (UniqueName: \"kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:47.974002 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:47.973947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.074906 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.074806    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.074906 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.074848    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.074906 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.074883    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrnc\" (UniqueName: \"kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.075206 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.074935    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.075206 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.074968    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.075407 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.075382    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.075618 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.075594    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.075618 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.075609    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.077391 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.077371    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.082265 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.082247    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrnc\" (UniqueName: \"kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.199033 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.198994    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:48.328475 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.328380    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:33:48.331327 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:33:48.331299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc13dcd5_0536_484d_b808_2ab218cc7bff.slice/crio-d1d84f324e205a93be972a2e9ce63d2fe7efbcd5e916e8780bebc76d7657f331 WatchSource:0}: Error finding container d1d84f324e205a93be972a2e9ce63d2fe7efbcd5e916e8780bebc76d7657f331: Status 404 returned error can't find the container with id d1d84f324e205a93be972a2e9ce63d2fe7efbcd5e916e8780bebc76d7657f331
Apr 17 17:33:48.419668 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.419640    2579 generic.go:358] "Generic (PLEG): container finished" podID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerID="368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a" exitCode=2
Apr 17 17:33:48.419887 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.419707    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerDied","Data":"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"}
Apr 17 17:33:48.421072 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.421045    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerStarted","Data":"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"}
Apr 17 17:33:48.421168 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:48.421078    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerStarted","Data":"d1d84f324e205a93be972a2e9ce63d2fe7efbcd5e916e8780bebc76d7657f331"}
Apr 17 17:33:49.426073 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:49.426040    2579 generic.go:358] "Generic (PLEG): container finished" podID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerID="c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1" exitCode=0
Apr 17 17:33:49.426463 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:49.426091    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerDied","Data":"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"}
Apr 17 17:33:50.431605 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:50.431576    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerStarted","Data":"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"}
Apr 17 17:33:50.431605 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:50.431609    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerStarted","Data":"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"}
Apr 17 17:33:50.432047 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:50.431716    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:50.449692 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:50.449633    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podStartSLOduration=3.449615583 podStartE2EDuration="3.449615583s" podCreationTimestamp="2026-04-17 17:33:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:33:50.448540509 +0000 UTC m=+3757.689198945" watchObservedRunningTime="2026-04-17 17:33:50.449615583 +0000 UTC m=+3757.690274019"
Apr 17 17:33:51.190553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:51.190505    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Apr 17 17:33:51.435630 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:51.435596    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:51.437115 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:51.437086    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:33:52.174055 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.174033    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:33:52.211533 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211501    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") "
Apr 17 17:33:52.211704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211542    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmx5t\" (UniqueName: \"kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t\") pod \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") "
Apr 17 17:33:52.211704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211585    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location\") pod \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") "
Apr 17 17:33:52.211704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211613    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls\") pod \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\" (UID: \"e7b3e9e3-76c5-4323-9c52-f90f47af846e\") "
Apr 17 17:33:52.211975 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211950    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "e7b3e9e3-76c5-4323-9c52-f90f47af846e" (UID: "e7b3e9e3-76c5-4323-9c52-f90f47af846e"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:33:52.212037 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.211952    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7b3e9e3-76c5-4323-9c52-f90f47af846e" (UID: "e7b3e9e3-76c5-4323-9c52-f90f47af846e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:33:52.213774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.213737    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t" (OuterVolumeSpecName: "kube-api-access-dmx5t") pod "e7b3e9e3-76c5-4323-9c52-f90f47af846e" (UID: "e7b3e9e3-76c5-4323-9c52-f90f47af846e"). InnerVolumeSpecName "kube-api-access-dmx5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:33:52.213879 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.213744    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7b3e9e3-76c5-4323-9c52-f90f47af846e" (UID: "e7b3e9e3-76c5-4323-9c52-f90f47af846e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:33:52.313030 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.312938    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:33:52.313030 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.312978    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b3e9e3-76c5-4323-9c52-f90f47af846e-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:33:52.313030 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.312989    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b3e9e3-76c5-4323-9c52-f90f47af846e-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:33:52.313030 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.313007    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmx5t\" (UniqueName: \"kubernetes.io/projected/e7b3e9e3-76c5-4323-9c52-f90f47af846e-kube-api-access-dmx5t\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:33:52.440385 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.440346    2579 generic.go:358] "Generic (PLEG): container finished" podID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerID="466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6" exitCode=0
Apr 17 17:33:52.440808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.440434    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"
Apr 17 17:33:52.440808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.440436    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerDied","Data":"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"}
Apr 17 17:33:52.440808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.440478    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m" event={"ID":"e7b3e9e3-76c5-4323-9c52-f90f47af846e","Type":"ContainerDied","Data":"7d4000ef55b2426aeb884ee8462c62df7956ebfb320eb492694eb4d8aa3afe50"}
Apr 17 17:33:52.440808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.440497    2579 scope.go:117] "RemoveContainer" containerID="368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"
Apr 17 17:33:52.441196 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.441168    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:33:52.448872 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.448844    2579 scope.go:117] "RemoveContainer" containerID="466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"
Apr 17 17:33:52.456111 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.456091    2579 scope.go:117] "RemoveContainer" containerID="b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"
Apr 17 17:33:52.463351 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.463330    2579 scope.go:117] "RemoveContainer" containerID="368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"
Apr 17 17:33:52.463644 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:33:52.463628    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a\": container with ID starting with 368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a not found: ID does not exist" containerID="368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"
Apr 17 17:33:52.463724 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.463656    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a"} err="failed to get container status \"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a\": rpc error: code = NotFound desc = could not find container \"368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a\": container with ID starting with 368c4123e0af6f5b35888cd8ea29574821ec7b8946473599aacb96a6f346a34a not found: ID does not exist"
Apr 17 17:33:52.463724 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.463679    2579 scope.go:117] "RemoveContainer" containerID="466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"
Apr 17 17:33:52.463948 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:33:52.463924    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6\": container with ID starting with 466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6 not found: ID does not exist" containerID="466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"
Apr 17 17:33:52.463990 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.463959    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6"} err="failed to get container status \"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6\": rpc error: code = NotFound desc = could not find container \"466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6\": container with ID starting with 466237c665f9d3e3c84b77975e3e12a98d97b0309afb9d0d6f1fd55798eff9d6 not found: ID does not exist"
Apr 17 17:33:52.463990 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.463980    2579 scope.go:117] "RemoveContainer" containerID="b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"
Apr 17 17:33:52.464230 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:33:52.464213    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a\": container with ID starting with b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a not found: ID does not exist" containerID="b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"
Apr 17 17:33:52.464302 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.464236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a"} err="failed to get container status \"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a\": rpc error: code = NotFound desc = could not find container \"b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a\": container with ID starting with b22a2cd823b1c8ee440cd49b86a9cd522166f316d383d85cd6bcd9cbfd753c9a not found: ID does not exist"
Apr 17 17:33:52.464656 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.464638    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:33:52.477182 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:52.477150    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-jcz4m"]
Apr 17 17:33:53.389598 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:53.389561    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" path="/var/lib/kubelet/pods/e7b3e9e3-76c5-4323-9c52-f90f47af846e/volumes"
Apr 17 17:33:57.445061 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:57.445032    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:33:57.445615 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:33:57.445590    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:07.446475 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:07.446432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:17.445863 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:17.445825    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:27.446421 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:27.446383    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:37.445922 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:37.445882    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:47.446456 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:47.446418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 17 17:34:57.445910 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:57.445874    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:34:57.878015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:57.877979    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:34:57.878373 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:57.878321    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container" containerID="cri-o://b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4" gracePeriod=30
Apr 17 17:34:57.878536 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:57.878372    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kube-rbac-proxy" containerID="cri-o://2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87" gracePeriod=30
Apr 17 17:34:58.662486 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.662453    2579 generic.go:358] "Generic (PLEG): container finished" podID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerID="2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87" exitCode=2
Apr 17 17:34:58.662861 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.662533    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerDied","Data":"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"}
Apr 17 17:34:58.944187 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944107    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:34:58.944483 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944471    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container"
Apr 17 17:34:58.944535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944485    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container"
Apr 17 17:34:58.944535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944497    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="storage-initializer"
Apr 17 17:34:58.944535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944504    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="storage-initializer"
Apr 17 17:34:58.944535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944525    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kube-rbac-proxy"
Apr 17 17:34:58.944535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944530    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kube-rbac-proxy"
Apr 17 17:34:58.944724 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944579    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kserve-container"
Apr 17 17:34:58.944724 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.944591    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="e7b3e9e3-76c5-4323-9c52-f90f47af846e" containerName="kube-rbac-proxy"
Apr 17 17:34:58.947740 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.947720    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:58.949780 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.949741    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Apr 17 17:34:58.949878 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.949749    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Apr 17 17:34:58.956311 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:58.956279    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:34:59.071536 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.071499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnlj\" (UniqueName: \"kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.071704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.071545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.071704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.071579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.071704 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.071627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.172278 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.172244    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.172473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.172302    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.172473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.172351    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnlj\" (UniqueName: \"kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.172473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.172379    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.172808 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.172786    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.173058 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.173039    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.174708 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.174688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.180598 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.180575    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnlj\" (UniqueName: \"kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.259348 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.259270    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:34:59.381727 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.381705    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:34:59.384673 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:34:59.384631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9035fd2_ffab_4ad5_899d_8f02d9c66079.slice/crio-4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775 WatchSource:0}: Error finding container 4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775: Status 404 returned error can't find the container with id 4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775
Apr 17 17:34:59.668130 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.668097    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerStarted","Data":"b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86"}
Apr 17 17:34:59.668130 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:34:59.668132    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerStarted","Data":"4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775"}
Apr 17 17:35:01.676431 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:01.676402    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/0.log"
Apr 17 17:35:01.676832 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:01.676440    2579 generic.go:358] "Generic (PLEG): container finished" podID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerID="b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86" exitCode=1
Apr 17 17:35:01.676832 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:01.676521    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerDied","Data":"b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86"}
Apr 17 17:35:02.232692 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.232662    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:35:02.296096 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296003    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls\") pod \"cc13dcd5-0536-484d-b808-2ab218cc7bff\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") "
Apr 17 17:35:02.296096 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296061    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"cc13dcd5-0536-484d-b808-2ab218cc7bff\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") "
Apr 17 17:35:02.296096 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296094    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqrnc\" (UniqueName: \"kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc\") pod \"cc13dcd5-0536-484d-b808-2ab218cc7bff\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") "
Apr 17 17:35:02.296400 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296116    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert\") pod \"cc13dcd5-0536-484d-b808-2ab218cc7bff\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") "
Apr 17 17:35:02.296400 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296223    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location\") pod \"cc13dcd5-0536-484d-b808-2ab218cc7bff\" (UID: \"cc13dcd5-0536-484d-b808-2ab218cc7bff\") "
Apr 17 17:35:02.296548 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296508    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc13dcd5-0536-484d-b808-2ab218cc7bff" (UID: "cc13dcd5-0536-484d-b808-2ab218cc7bff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:35:02.296601 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296523    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "cc13dcd5-0536-484d-b808-2ab218cc7bff" (UID: "cc13dcd5-0536-484d-b808-2ab218cc7bff"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:35:02.296601 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.296530    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc13dcd5-0536-484d-b808-2ab218cc7bff" (UID: "cc13dcd5-0536-484d-b808-2ab218cc7bff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:35:02.298351 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.298329    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc" (OuterVolumeSpecName: "kube-api-access-sqrnc") pod "cc13dcd5-0536-484d-b808-2ab218cc7bff" (UID: "cc13dcd5-0536-484d-b808-2ab218cc7bff"). InnerVolumeSpecName "kube-api-access-sqrnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:35:02.298430 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.298347    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc13dcd5-0536-484d-b808-2ab218cc7bff" (UID: "cc13dcd5-0536-484d-b808-2ab218cc7bff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:35:02.397495 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.397458    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc13dcd5-0536-484d-b808-2ab218cc7bff-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:02.397495 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.397494    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:02.397720 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.397509    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqrnc\" (UniqueName: \"kubernetes.io/projected/cc13dcd5-0536-484d-b808-2ab218cc7bff-kube-api-access-sqrnc\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:02.397720 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.397524    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc13dcd5-0536-484d-b808-2ab218cc7bff-cabundle-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:02.397720 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.397536    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc13dcd5-0536-484d-b808-2ab218cc7bff-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:02.681170 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.681137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/0.log"
Apr 17 17:35:02.681622 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.681219    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerStarted","Data":"644de8a9645078ef556860bdc1da6188b039e413592137dae486a90b5d279ab6"}
Apr 17 17:35:02.682726 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.682698    2579 generic.go:358] "Generic (PLEG): container finished" podID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerID="b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4" exitCode=0
Apr 17 17:35:02.682891 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.682792    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerDied","Data":"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"}
Apr 17 17:35:02.682891 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.682824    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"
Apr 17 17:35:02.682891 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.682838    2579 scope.go:117] "RemoveContainer" containerID="2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"
Apr 17 17:35:02.683026 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.682828    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x" event={"ID":"cc13dcd5-0536-484d-b808-2ab218cc7bff","Type":"ContainerDied","Data":"d1d84f324e205a93be972a2e9ce63d2fe7efbcd5e916e8780bebc76d7657f331"}
Apr 17 17:35:02.690786 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.690748    2579 scope.go:117] "RemoveContainer" containerID="b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"
Apr 17 17:35:02.697925 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.697908    2579 scope.go:117] "RemoveContainer" containerID="c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"
Apr 17 17:35:02.706166 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.706145    2579 scope.go:117] "RemoveContainer" containerID="2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"
Apr 17 17:35:02.706440 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:02.706422    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87\": container with ID starting with 2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87 not found: ID does not exist" containerID="2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"
Apr 17 17:35:02.706514 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.706449    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87"} err="failed to get container status \"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87\": rpc error: code = NotFound desc = could not find container \"2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87\": container with ID starting with 2d98b8db76df5092fd930672b509e5524b3e9c29fafac75b86c12582ea3c8b87 not found: ID does not exist"
Apr 17 17:35:02.706514 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.706468    2579 scope.go:117] "RemoveContainer" containerID="b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"
Apr 17 17:35:02.706740 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:02.706722    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4\": container with ID starting with b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4 not found: ID does not exist" containerID="b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"
Apr 17 17:35:02.706817 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.706748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4"} err="failed to get container status \"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4\": rpc error: code = NotFound desc = could not find container \"b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4\": container with ID starting with b9dc1c8eca963c97e36ac9e37915feac2c002d0f62edff36e825809da67978a4 not found: ID does not exist"
Apr 17 17:35:02.706817 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.706783    2579 scope.go:117] "RemoveContainer" containerID="c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"
Apr 17 17:35:02.707063 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:02.707046    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1\": container with ID starting with c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1 not found: ID does not exist" containerID="c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"
Apr 17 17:35:02.707148 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.707066    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1"} err="failed to get container status \"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1\": rpc error: code = NotFound desc = could not find container \"c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1\": container with ID starting with c4a1d0dadeead1dcf6ece602d032170b6e6774b56d45d32e380aa47a9af18fb1 not found: ID does not exist"
Apr 17 17:35:02.712733 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.712702    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:35:02.714434 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:02.714411    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-czh6x"]
Apr 17 17:35:03.389170 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:03.389138    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" path="/var/lib/kubelet/pods/cc13dcd5-0536-484d-b808-2ab218cc7bff/volumes"
Apr 17 17:35:04.691955 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.691929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/1.log"
Apr 17 17:35:04.692343 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.692272    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/0.log"
Apr 17 17:35:04.692343 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.692306    2579 generic.go:358] "Generic (PLEG): container finished" podID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerID="644de8a9645078ef556860bdc1da6188b039e413592137dae486a90b5d279ab6" exitCode=1
Apr 17 17:35:04.692432 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.692341    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerDied","Data":"644de8a9645078ef556860bdc1da6188b039e413592137dae486a90b5d279ab6"}
Apr 17 17:35:04.692432 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.692372    2579 scope.go:117] "RemoveContainer" containerID="b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86"
Apr 17 17:35:04.692775 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:04.692747    2579 scope.go:117] "RemoveContainer" containerID="b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86"
Apr 17 17:35:04.702810 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:04.702781    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_kserve-ci-e2e-test_c9035fd2-ffab-4ad5-899d-8f02d9c66079_0 in pod sandbox 4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775 from index: no such id: 'b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86'" containerID="b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86"
Apr 17 17:35:04.702907 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:04.702826    2579 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_kserve-ci-e2e-test_c9035fd2-ffab-4ad5-899d-8f02d9c66079_0 in pod sandbox 4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775 from index: no such id: 'b8a17fbfc3aabb37f4d7263c71da830c9777ae9f344a89ca435cadf06e079a86'; Skipping pod \"isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_kserve-ci-e2e-test(c9035fd2-ffab-4ad5-899d-8f02d9c66079)\"" logger="UnhandledError"
Apr 17 17:35:04.704168 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:35:04.704144    2579 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_kserve-ci-e2e-test(c9035fd2-ffab-4ad5-899d-8f02d9c66079)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079"
Apr 17 17:35:05.696536 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:05.696507    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/1.log"
Apr 17 17:35:08.940672 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:08.940637    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:35:09.073605 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.073584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/1.log"
Apr 17 17:35:09.073708 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.073646    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:35:09.157377 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157344    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls\") pod \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") "
Apr 17 17:35:09.157526 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157391    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfnlj\" (UniqueName: \"kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj\") pod \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") "
Apr 17 17:35:09.157526 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157436    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location\") pod \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") "
Apr 17 17:35:09.157526 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157504    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\" (UID: \"c9035fd2-ffab-4ad5-899d-8f02d9c66079\") "
Apr 17 17:35:09.157698 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157671    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9035fd2-ffab-4ad5-899d-8f02d9c66079" (UID: "c9035fd2-ffab-4ad5-899d-8f02d9c66079"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:35:09.157784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157740    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:09.157907 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.157883    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "c9035fd2-ffab-4ad5-899d-8f02d9c66079" (UID: "c9035fd2-ffab-4ad5-899d-8f02d9c66079"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:35:09.159453 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.159419    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9035fd2-ffab-4ad5-899d-8f02d9c66079" (UID: "c9035fd2-ffab-4ad5-899d-8f02d9c66079"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:35:09.159572 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.159498    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj" (OuterVolumeSpecName: "kube-api-access-mfnlj") pod "c9035fd2-ffab-4ad5-899d-8f02d9c66079" (UID: "c9035fd2-ffab-4ad5-899d-8f02d9c66079"). InnerVolumeSpecName "kube-api-access-mfnlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:35:09.259032 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.258966    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9035fd2-ffab-4ad5-899d-8f02d9c66079-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:09.259032 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.258991    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9035fd2-ffab-4ad5-899d-8f02d9c66079-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:09.259032 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.259002    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfnlj\" (UniqueName: \"kubernetes.io/projected/c9035fd2-ffab-4ad5-899d-8f02d9c66079-kube-api-access-mfnlj\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:35:09.710244 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.710217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x_c9035fd2-ffab-4ad5-899d-8f02d9c66079/storage-initializer/1.log"
Apr 17 17:35:09.710403 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.710349    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"
Apr 17 17:35:09.710403 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.710349    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x" event={"ID":"c9035fd2-ffab-4ad5-899d-8f02d9c66079","Type":"ContainerDied","Data":"4cf2f1af9ff613b51700c55a35a515a5215cf673c4e70e73ca66c8ab25086775"}
Apr 17 17:35:09.710403 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.710392    2579 scope.go:117] "RemoveContainer" containerID="644de8a9645078ef556860bdc1da6188b039e413592137dae486a90b5d279ab6"
Apr 17 17:35:09.738198 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.738173    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:35:09.741410 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.741385    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-qvq2x"]
Apr 17 17:35:09.988400 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988305    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988679    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988691    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988707    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kube-rbac-proxy"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988712    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kube-rbac-proxy"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988721    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="storage-initializer"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988729    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="storage-initializer"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988739    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988746    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988756    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.988774 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988772    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.989088 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988842    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kserve-container"
Apr 17 17:35:09.989088 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988851    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="cc13dcd5-0536-484d-b808-2ab218cc7bff" containerName="kube-rbac-proxy"
Apr 17 17:35:09.989088 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988858    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.989088 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.988968    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" containerName="storage-initializer"
Apr 17 17:35:09.993926 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.993903    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:09.996119 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.996090    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Apr 17 17:35:09.996119 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.996113    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 17 17:35:09.996305 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.996163    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 17 17:35:09.997309 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.997289    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Apr 17 17:35:09.997478 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.997458    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wzvpv\""
Apr 17 17:35:09.997615 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.997338    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 17 17:35:09.997896 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.997405    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Apr 17 17:35:09.999161 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:09.999140    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:35:10.066133 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.066097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.066286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.066148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.066286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.066167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.066286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.066213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwlw\" (UniqueName: \"kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.066421 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.066309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.166804 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.166747    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.166966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.166863    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.166966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.166906    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.166966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.166924    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.166966 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.166941    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwlw\" (UniqueName: \"kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.167232 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.167164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.167543 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.167522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.167627 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.167546    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.169474 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.169449    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.175478 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.175434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwlw\" (UniqueName: \"kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.307045 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.306956    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:10.424336 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.424305    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:35:10.427072 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:35:10.427043    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f9ec72_296c_489c_b8a1_b4f01db5d65c.slice/crio-73a780e8e5e2b6fb7841f3bb3fb4b0c3a1b48ab93ad68cb6d8d57fc0621253fa WatchSource:0}: Error finding container 73a780e8e5e2b6fb7841f3bb3fb4b0c3a1b48ab93ad68cb6d8d57fc0621253fa: Status 404 returned error can't find the container with id 73a780e8e5e2b6fb7841f3bb3fb4b0c3a1b48ab93ad68cb6d8d57fc0621253fa
Apr 17 17:35:10.716518 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.716486    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerStarted","Data":"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"}
Apr 17 17:35:10.716518 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:10.716523    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerStarted","Data":"73a780e8e5e2b6fb7841f3bb3fb4b0c3a1b48ab93ad68cb6d8d57fc0621253fa"}
Apr 17 17:35:11.389017 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:11.388987    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9035fd2-ffab-4ad5-899d-8f02d9c66079" path="/var/lib/kubelet/pods/c9035fd2-ffab-4ad5-899d-8f02d9c66079/volumes"
Apr 17 17:35:11.721874 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:11.721842    2579 generic.go:358] "Generic (PLEG): container finished" podID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerID="be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9" exitCode=0
Apr 17 17:35:11.722075 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:11.721931    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerDied","Data":"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"}
Apr 17 17:35:12.727483 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:12.727413    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerStarted","Data":"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"}
Apr 17 17:35:12.727483 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:12.727487    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerStarted","Data":"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"}
Apr 17 17:35:12.727901 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:12.727642    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:12.748687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:12.748644    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podStartSLOduration=3.74863203 podStartE2EDuration="3.74863203s" podCreationTimestamp="2026-04-17 17:35:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:35:12.74645889 +0000 UTC m=+3839.987117363" watchObservedRunningTime="2026-04-17 17:35:12.74863203 +0000 UTC m=+3839.989290466"
Apr 17 17:35:13.732453 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:13.732427    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:13.733705 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:13.733681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:14.735830 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:14.735793    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:19.739990 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:19.739959    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:35:19.740523 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:19.740494    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:29.741081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:29.740998    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:39.741177 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:39.741139    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:49.740553 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:49.740512    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:35:59.740452 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:35:59.740413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:36:09.741114 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:09.741075    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 17 17:36:13.620992 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:13.620959    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:36:13.623981 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:13.623960    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:36:13.624508 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:13.624489    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:36:13.627383 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:13.627362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:36:19.740926 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:19.740893    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:36:20.030989 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:20.030905    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:36:20.031216 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:20.031186    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container" containerID="cri-o://5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12" gracePeriod=30
Apr 17 17:36:20.031363 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:20.031223    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kube-rbac-proxy" containerID="cri-o://10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596" gracePeriod=30
Apr 17 17:36:20.959516 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:20.959483    2579 generic.go:358] "Generic (PLEG): container finished" podID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerID="10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596" exitCode=2
Apr 17 17:36:20.960001 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:20.959541    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerDied","Data":"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"}
Apr 17 17:36:21.121411 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.121368    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:21.125590 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.125567    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.127827 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.127803    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Apr 17 17:36:21.127942 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.127826    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Apr 17 17:36:21.132861 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.132837    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:21.264278 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.264186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422wz\" (UniqueName: \"kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.264278 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.264229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.264278 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.264254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.264524 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.264392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.365415 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.365373    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-422wz\" (UniqueName: \"kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.365595 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.365416    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.365595 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.365454    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.365595 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.365493    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.366002 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.365978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.366192 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.366174    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.368015 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.367994    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.375370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.375348    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-422wz\" (UniqueName: \"kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.438329 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.438297    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:21.566266 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.566237    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:21.568540 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:36:21.568509    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1167b39_d155_4cee_b23c_c47db3a14e11.slice/crio-9dac2cd3c3b8784f001a367462f5f49e6d06381531e00d70b6ab08c06ea01f31 WatchSource:0}: Error finding container 9dac2cd3c3b8784f001a367462f5f49e6d06381531e00d70b6ab08c06ea01f31: Status 404 returned error can't find the container with id 9dac2cd3c3b8784f001a367462f5f49e6d06381531e00d70b6ab08c06ea01f31
Apr 17 17:36:21.964889 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.964849    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerStarted","Data":"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"}
Apr 17 17:36:21.964889 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:21.964886    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerStarted","Data":"9dac2cd3c3b8784f001a367462f5f49e6d06381531e00d70b6ab08c06ea01f31"}
Apr 17 17:36:24.579136 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.579115    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:36:24.695606 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.695500    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") "
Apr 17 17:36:24.695606 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.695581    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nwlw\" (UniqueName: \"kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw\") pod \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") "
Apr 17 17:36:24.695854 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.695666    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls\") pod \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") "
Apr 17 17:36:24.695854 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.695699    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert\") pod \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") "
Apr 17 17:36:24.695854 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.695741    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location\") pod \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\" (UID: \"91f9ec72-296c-489c-b8a1-b4f01db5d65c\") "
Apr 17 17:36:24.696098 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.696072    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "91f9ec72-296c-489c-b8a1-b4f01db5d65c" (UID: "91f9ec72-296c-489c-b8a1-b4f01db5d65c"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:36:24.696183 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.696101    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91f9ec72-296c-489c-b8a1-b4f01db5d65c" (UID: "91f9ec72-296c-489c-b8a1-b4f01db5d65c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:36:24.696183 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.696162    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91f9ec72-296c-489c-b8a1-b4f01db5d65c" (UID: "91f9ec72-296c-489c-b8a1-b4f01db5d65c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:36:24.697733 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.697706    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw" (OuterVolumeSpecName: "kube-api-access-2nwlw") pod "91f9ec72-296c-489c-b8a1-b4f01db5d65c" (UID: "91f9ec72-296c-489c-b8a1-b4f01db5d65c"). InnerVolumeSpecName "kube-api-access-2nwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:36:24.697839 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.697731    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91f9ec72-296c-489c-b8a1-b4f01db5d65c" (UID: "91f9ec72-296c-489c-b8a1-b4f01db5d65c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:36:24.796621 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.796579    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-cabundle-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:24.796621 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.796616    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:24.796845 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.796634    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f9ec72-296c-489c-b8a1-b4f01db5d65c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:24.796845 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.796649    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwlw\" (UniqueName: \"kubernetes.io/projected/91f9ec72-296c-489c-b8a1-b4f01db5d65c-kube-api-access-2nwlw\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:24.796845 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.796662    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f9ec72-296c-489c-b8a1-b4f01db5d65c-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:24.976513 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.976422    2579 generic.go:358] "Generic (PLEG): container finished" podID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerID="5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12" exitCode=0
Apr 17 17:36:24.976687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.976509    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerDied","Data":"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"}
Apr 17 17:36:24.976687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.976545    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"
Apr 17 17:36:24.976687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.976561    2579 scope.go:117] "RemoveContainer" containerID="10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"
Apr 17 17:36:24.976687 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.976550    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v" event={"ID":"91f9ec72-296c-489c-b8a1-b4f01db5d65c","Type":"ContainerDied","Data":"73a780e8e5e2b6fb7841f3bb3fb4b0c3a1b48ab93ad68cb6d8d57fc0621253fa"}
Apr 17 17:36:24.984742 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.984725    2579 scope.go:117] "RemoveContainer" containerID="5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"
Apr 17 17:36:24.991811 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.991790    2579 scope.go:117] "RemoveContainer" containerID="be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"
Apr 17 17:36:24.997976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.997956    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:36:24.998956 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.998939    2579 scope.go:117] "RemoveContainer" containerID="10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"
Apr 17 17:36:24.999196 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:24.999175    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596\": container with ID starting with 10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596 not found: ID does not exist" containerID="10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"
Apr 17 17:36:24.999284 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.999202    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596"} err="failed to get container status \"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596\": rpc error: code = NotFound desc = could not find container \"10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596\": container with ID starting with 10d7c1951f3446ab2870f603c44b476596099897ff046df1a2443138a2159596 not found: ID does not exist"
Apr 17 17:36:24.999284 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.999221    2579 scope.go:117] "RemoveContainer" containerID="5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"
Apr 17 17:36:24.999459 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:24.999443    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12\": container with ID starting with 5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12 not found: ID does not exist" containerID="5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"
Apr 17 17:36:24.999552 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.999465    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12"} err="failed to get container status \"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12\": rpc error: code = NotFound desc = could not find container \"5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12\": container with ID starting with 5806bd8896082d65964df24913a9c116240792d98b2e21140aaadc441c826b12 not found: ID does not exist"
Apr 17 17:36:24.999552 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.999480    2579 scope.go:117] "RemoveContainer" containerID="be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"
Apr 17 17:36:24.999747 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:24.999722    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9\": container with ID starting with be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9 not found: ID does not exist" containerID="be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"
Apr 17 17:36:24.999845 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:24.999756    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9"} err="failed to get container status \"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9\": rpc error: code = NotFound desc = could not find container \"be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9\": container with ID starting with be2862816231b82f27c82544dce4f2e5bee8cb499429a810cc1add7d2acab8c9 not found: ID does not exist"
Apr 17 17:36:25.001695 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:25.001676    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-c8k2v"]
Apr 17 17:36:25.389350 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:25.389315    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" path="/var/lib/kubelet/pods/91f9ec72-296c-489c-b8a1-b4f01db5d65c/volumes"
Apr 17 17:36:27.986959 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:27.986934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/0.log"
Apr 17 17:36:27.987354 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:27.986969    2579 generic.go:358] "Generic (PLEG): container finished" podID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerID="d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb" exitCode=1
Apr 17 17:36:27.987354 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:27.986994    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerDied","Data":"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"}
Apr 17 17:36:28.991610 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:28.991585    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/0.log"
Apr 17 17:36:28.992008 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:28.991668    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerStarted","Data":"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"}
Apr 17 17:36:31.114121 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:31.114076    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:31.114703 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:31.114384    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer" containerID="cri-o://aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a" gracePeriod=30
Apr 17 17:36:32.175317 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175275    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:36:32.175846 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175825    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container"
Apr 17 17:36:32.175930 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175849    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container"
Apr 17 17:36:32.175930 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175880    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="storage-initializer"
Apr 17 17:36:32.175930 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="storage-initializer"
Apr 17 17:36:32.175930 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175906    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kube-rbac-proxy"
Apr 17 17:36:32.175930 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.175914    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kube-rbac-proxy"
Apr 17 17:36:32.176200 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.176001    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kserve-container"
Apr 17 17:36:32.176200 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.176019    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="91f9ec72-296c-489c-b8a1-b4f01db5d65c" containerName="kube-rbac-proxy"
Apr 17 17:36:32.179610 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.179588    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.181976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.181956    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Apr 17 17:36:32.182080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.181958    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 17 17:36:32.182080 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.181999    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Apr 17 17:36:32.189684 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.189659    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:36:32.260672 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.260641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.260885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.260712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmw7\" (UniqueName: \"kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.260885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.260737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.260885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.260781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.260885 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.260799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.361685 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.361653    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmw7\" (UniqueName: \"kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.361862 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.361694    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.361862 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.361752    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.361862 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.361795    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.361862 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.361829    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.362052 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:32.361983    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Apr 17 17:36:32.362102 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:32.362069    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls podName:090c7a0e-ff9c-41c0-9150-d49a52f54dcc nodeName:}" failed. No retries permitted until 2026-04-17 17:36:32.862047135 +0000 UTC m=+3920.102705563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Apr 17 17:36:32.362189 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.362166    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.362451 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.362435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.362491 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.362438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.370679 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.370662    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmw7\" (UniqueName: \"kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.866505 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.866472    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.868964 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.868943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:32.958606 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.958585    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/1.log"
Apr 17 17:36:32.958912 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.958900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/0.log"
Apr 17 17:36:32.958972 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.958956    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:32.967117 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967094    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls\") pod \"b1167b39-d155-4cee-b23c-c47db3a14e11\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") "
Apr 17 17:36:32.967230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967166    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location\") pod \"b1167b39-d155-4cee-b23c-c47db3a14e11\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") "
Apr 17 17:36:32.967230 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967199    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"b1167b39-d155-4cee-b23c-c47db3a14e11\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") "
Apr 17 17:36:32.967349 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967241    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-422wz\" (UniqueName: \"kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz\") pod \"b1167b39-d155-4cee-b23c-c47db3a14e11\" (UID: \"b1167b39-d155-4cee-b23c-c47db3a14e11\") "
Apr 17 17:36:32.967429 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967397    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1167b39-d155-4cee-b23c-c47db3a14e11" (UID: "b1167b39-d155-4cee-b23c-c47db3a14e11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:36:32.967587 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967566    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "b1167b39-d155-4cee-b23c-c47db3a14e11" (UID: "b1167b39-d155-4cee-b23c-c47db3a14e11"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:36:32.967651 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.967596    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1167b39-d155-4cee-b23c-c47db3a14e11-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:32.969131 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.969112    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1167b39-d155-4cee-b23c-c47db3a14e11" (UID: "b1167b39-d155-4cee-b23c-c47db3a14e11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:36:32.969214 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:32.969195    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz" (OuterVolumeSpecName: "kube-api-access-422wz") pod "b1167b39-d155-4cee-b23c-c47db3a14e11" (UID: "b1167b39-d155-4cee-b23c-c47db3a14e11"). InnerVolumeSpecName "kube-api-access-422wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:36:33.004953 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.004930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/1.log"
Apr 17 17:36:33.005318 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005304    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb_b1167b39-d155-4cee-b23c-c47db3a14e11/storage-initializer/0.log"
Apr 17 17:36:33.005369 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005337    2579 generic.go:358] "Generic (PLEG): container finished" podID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerID="aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a" exitCode=1
Apr 17 17:36:33.005410 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005392    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerDied","Data":"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"}
Apr 17 17:36:33.005446 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005414    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb" event={"ID":"b1167b39-d155-4cee-b23c-c47db3a14e11","Type":"ContainerDied","Data":"9dac2cd3c3b8784f001a367462f5f49e6d06381531e00d70b6ab08c06ea01f31"}
Apr 17 17:36:33.005446 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005417    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"
Apr 17 17:36:33.005446 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.005428    2579 scope.go:117] "RemoveContainer" containerID="aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"
Apr 17 17:36:33.013832 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.013809    2579 scope.go:117] "RemoveContainer" containerID="d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"
Apr 17 17:36:33.020807 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.020792    2579 scope.go:117] "RemoveContainer" containerID="aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"
Apr 17 17:36:33.021053 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:33.021035    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a\": container with ID starting with aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a not found: ID does not exist" containerID="aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"
Apr 17 17:36:33.021105 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.021061    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a"} err="failed to get container status \"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a\": rpc error: code = NotFound desc = could not find container \"aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a\": container with ID starting with aaaee64970c4d94fe34963b6ccf713ff797a92b06c56a56d0fca31f9aad5471a not found: ID does not exist"
Apr 17 17:36:33.021105 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.021078    2579 scope.go:117] "RemoveContainer" containerID="d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"
Apr 17 17:36:33.021307 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:36:33.021290    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb\": container with ID starting with d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb not found: ID does not exist" containerID="d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"
Apr 17 17:36:33.021352 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.021313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb"} err="failed to get container status \"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb\": rpc error: code = NotFound desc = could not find container \"d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb\": container with ID starting with d3e22c8173be433856f711337840ab238329af2c29a54dc390d7ecb5c05b4afb not found: ID does not exist"
Apr 17 17:36:33.046974 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.046949    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:33.050486 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.050465    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-hqrvb"]
Apr 17 17:36:33.068473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.068418    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1167b39-d155-4cee-b23c-c47db3a14e11-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:33.068473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.068440    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422wz\" (UniqueName: \"kubernetes.io/projected/b1167b39-d155-4cee-b23c-c47db3a14e11-kube-api-access-422wz\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:33.068473 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.068450    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1167b39-d155-4cee-b23c-c47db3a14e11-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:36:33.091455 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.091435    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:33.220086 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.220061    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:36:33.222243 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:36:33.222217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090c7a0e_ff9c_41c0_9150_d49a52f54dcc.slice/crio-35cc00f1e2fda92c1217ddeab74b5bdcfac614ebae2f7dcaa1dfc2775f6df243 WatchSource:0}: Error finding container 35cc00f1e2fda92c1217ddeab74b5bdcfac614ebae2f7dcaa1dfc2775f6df243: Status 404 returned error can't find the container with id 35cc00f1e2fda92c1217ddeab74b5bdcfac614ebae2f7dcaa1dfc2775f6df243
Apr 17 17:36:33.390612 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:33.390579    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" path="/var/lib/kubelet/pods/b1167b39-d155-4cee-b23c-c47db3a14e11/volumes"
Apr 17 17:36:34.010360 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:34.010324    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerStarted","Data":"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"}
Apr 17 17:36:34.010538 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:34.010370    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerStarted","Data":"35cc00f1e2fda92c1217ddeab74b5bdcfac614ebae2f7dcaa1dfc2775f6df243"}
Apr 17 17:36:35.015369 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:35.015336    2579 generic.go:358] "Generic (PLEG): container finished" podID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerID="2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5" exitCode=0
Apr 17 17:36:35.015890 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:35.015429    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerDied","Data":"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"}
Apr 17 17:36:36.021076 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:36.021042    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerStarted","Data":"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"}
Apr 17 17:36:36.021076 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:36.021078    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerStarted","Data":"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"}
Apr 17 17:36:36.021501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:36.021201    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:36.040011 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:36.039953    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podStartSLOduration=4.039934468 podStartE2EDuration="4.039934468s" podCreationTimestamp="2026-04-17 17:36:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:36:36.039466753 +0000 UTC m=+3923.280125189" watchObservedRunningTime="2026-04-17 17:36:36.039934468 +0000 UTC m=+3923.280592907"
Apr 17 17:36:37.024837 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:37.024800    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:37.026214 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:37.026189    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:36:38.027662 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:38.027622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:36:43.033176 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:43.033147    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:36:43.033612 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:43.033587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:36:53.033924 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:36:53.033883    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:03.034295 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:03.034256    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:13.034331 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:13.034293    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:23.034294 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:23.034251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:33.034603 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:33.034563    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:43.034613 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:43.034580    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:37:52.218216 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:52.218183    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:37:52.218701 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:52.218551    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" containerID="cri-o://57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0" gracePeriod=30
Apr 17 17:37:52.218701 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:52.218573    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kube-rbac-proxy" containerID="cri-o://ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd" gracePeriod=30
Apr 17 17:37:53.028221 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.028183    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Apr 17 17:37:53.034121 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.034094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 17 17:37:53.282836 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.282725    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283104    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283120    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283186    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.283322 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.283286    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1167b39-d155-4cee-b23c-c47db3a14e11" containerName="storage-initializer"
Apr 17 17:37:53.286279 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.286257    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.288815 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.288792    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Apr 17 17:37:53.288815 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.288814    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Apr 17 17:37:53.291857 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.291835    2579 generic.go:358] "Generic (PLEG): container finished" podID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerID="ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd" exitCode=2
Apr 17 17:37:53.291986 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.291897    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerDied","Data":"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"}
Apr 17 17:37:53.296730 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.296705    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:37:53.326104 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.326065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.326264 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.326127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdjc\" (UniqueName: \"kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.326264 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.326161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.326375 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.326264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.427020 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.426978    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.427212 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.427035    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdjc\" (UniqueName: \"kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.427212 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.427073    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.427212 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.427141    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.427212 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:37:53.427144    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Apr 17 17:37:53.427445 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:37:53.427273    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls podName:f43f1f30-376c-409d-aaeb-da92b67441c2 nodeName:}" failed. No retries permitted until 2026-04-17 17:37:53.927249122 +0000 UTC m=+4001.167907538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" (UID: "f43f1f30-376c-409d-aaeb-da92b67441c2") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Apr 17 17:37:53.432784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.428036    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.432784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.428470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.437084 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.437050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdjc\" (UniqueName: \"kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.930904 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.930869    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:53.933293 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:53.933272    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:54.199079 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:54.198992    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:37:54.325413 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:54.325388    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:37:54.327796 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:37:54.327747    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f1f30_376c_409d_aaeb_da92b67441c2.slice/crio-3c97e2c5d1fb84d51cfdce5c1faa3e4ae47b88fd0c007b7a0ae7f6df04b04830 WatchSource:0}: Error finding container 3c97e2c5d1fb84d51cfdce5c1faa3e4ae47b88fd0c007b7a0ae7f6df04b04830: Status 404 returned error can't find the container with id 3c97e2c5d1fb84d51cfdce5c1faa3e4ae47b88fd0c007b7a0ae7f6df04b04830
Apr 17 17:37:54.329530 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:54.329514    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 17 17:37:55.302195 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:55.302161    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerStarted","Data":"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"}
Apr 17 17:37:55.302195 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:55.302199    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerStarted","Data":"3c97e2c5d1fb84d51cfdce5c1faa3e4ae47b88fd0c007b7a0ae7f6df04b04830"}
Apr 17 17:37:56.468684 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.468663    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:37:56.555189 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555161    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") "
Apr 17 17:37:56.555189 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555190    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") pod \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") "
Apr 17 17:37:56.555394 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555237    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location\") pod \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") "
Apr 17 17:37:56.555394 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555270    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert\") pod \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") "
Apr 17 17:37:56.555394 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555312    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgmw7\" (UniqueName: \"kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7\") pod \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\" (UID: \"090c7a0e-ff9c-41c0-9150-d49a52f54dcc\") "
Apr 17 17:37:56.555686 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555661    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "090c7a0e-ff9c-41c0-9150-d49a52f54dcc" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:37:56.555819 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555692    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "090c7a0e-ff9c-41c0-9150-d49a52f54dcc" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:37:56.555819 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.555737    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "090c7a0e-ff9c-41c0-9150-d49a52f54dcc" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:37:56.557370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.557346    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "090c7a0e-ff9c-41c0-9150-d49a52f54dcc" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:37:56.557456 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.557430    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7" (OuterVolumeSpecName: "kube-api-access-xgmw7") pod "090c7a0e-ff9c-41c0-9150-d49a52f54dcc" (UID: "090c7a0e-ff9c-41c0-9150-d49a52f54dcc"). InnerVolumeSpecName "kube-api-access-xgmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:37:56.656074 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.656043    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmw7\" (UniqueName: \"kubernetes.io/projected/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kube-api-access-xgmw7\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:37:56.656074 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.656071    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:37:56.656264 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.656083    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:37:56.656264 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.656092    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:37:56.656264 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:56.656101    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090c7a0e-ff9c-41c0-9150-d49a52f54dcc-cabundle-cert\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:37:57.310850 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.310813    2579 generic.go:358] "Generic (PLEG): container finished" podID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerID="57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0" exitCode=0
Apr 17 17:37:57.311039 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.310857    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerDied","Data":"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"}
Apr 17 17:37:57.311039 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.310881    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps" event={"ID":"090c7a0e-ff9c-41c0-9150-d49a52f54dcc","Type":"ContainerDied","Data":"35cc00f1e2fda92c1217ddeab74b5bdcfac614ebae2f7dcaa1dfc2775f6df243"}
Apr 17 17:37:57.311039 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.310895    2579 scope.go:117] "RemoveContainer" containerID="ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"
Apr 17 17:37:57.311039 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.310895    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"
Apr 17 17:37:57.319630 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.319614    2579 scope.go:117] "RemoveContainer" containerID="57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"
Apr 17 17:37:57.326731 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.326711    2579 scope.go:117] "RemoveContainer" containerID="2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"
Apr 17 17:37:57.332076 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.332049    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:37:57.334977 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.334956    2579 scope.go:117] "RemoveContainer" containerID="ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"
Apr 17 17:37:57.335219 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:37:57.335200    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd\": container with ID starting with ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd not found: ID does not exist" containerID="ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"
Apr 17 17:37:57.335294 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.335227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd"} err="failed to get container status \"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd\": rpc error: code = NotFound desc = could not find container \"ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd\": container with ID starting with ef02f3a18d4b33c1157c93d6e62fdc799c49bc1c67ebc52640f9be3f162bbadd not found: ID does not exist"
Apr 17 17:37:57.335294 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.335244    2579 scope.go:117] "RemoveContainer" containerID="57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"
Apr 17 17:37:57.335477 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:37:57.335462    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0\": container with ID starting with 57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0 not found: ID does not exist" containerID="57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"
Apr 17 17:37:57.335535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.335480    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0"} err="failed to get container status \"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0\": rpc error: code = NotFound desc = could not find container \"57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0\": container with ID starting with 57c214ce7ad3447fb260def89329bde330e84b086cb2e44fd0d26f5197c50df0 not found: ID does not exist"
Apr 17 17:37:57.335535 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.335494    2579 scope.go:117] "RemoveContainer" containerID="2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"
Apr 17 17:37:57.335719 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:37:57.335703    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5\": container with ID starting with 2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5 not found: ID does not exist" containerID="2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"
Apr 17 17:37:57.335800 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.335725    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5"} err="failed to get container status \"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5\": rpc error: code = NotFound desc = could not find container \"2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5\": container with ID starting with 2c5f81f9bde6752b90d919c9543b2c85b89c415ff7b3c704c2be353929e4bea5 not found: ID does not exist"
Apr 17 17:37:57.336258 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.336240    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-48nps"]
Apr 17 17:37:57.389238 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:57.389212    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" path="/var/lib/kubelet/pods/090c7a0e-ff9c-41c0-9150-d49a52f54dcc/volumes"
Apr 17 17:37:59.320501 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:59.320467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/0.log"
Apr 17 17:37:59.320961 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:59.320512    2579 generic.go:358] "Generic (PLEG): container finished" podID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerID="4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7" exitCode=1
Apr 17 17:37:59.320961 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:37:59.320588    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerDied","Data":"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"}
Apr 17 17:38:00.325536 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:00.325507    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/0.log"
Apr 17 17:38:00.325972 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:00.325566    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerStarted","Data":"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"}
Apr 17 17:38:03.293906 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.293858    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:38:03.294414 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.294142    2579 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerName="storage-initializer" containerID="cri-o://d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2" gracePeriod=30
Apr 17 17:38:03.540018 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.539993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/1.log"
Apr 17 17:38:03.540321 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.540306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/0.log"
Apr 17 17:38:03.540382 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.540371    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:38:03.617877 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.617844    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"f43f1f30-376c-409d-aaeb-da92b67441c2\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") "
Apr 17 17:38:03.618086 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.617895    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") pod \"f43f1f30-376c-409d-aaeb-da92b67441c2\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") "
Apr 17 17:38:03.618086 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.617923    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvdjc\" (UniqueName: \"kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc\") pod \"f43f1f30-376c-409d-aaeb-da92b67441c2\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") "
Apr 17 17:38:03.618086 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.617955    2579 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location\") pod \"f43f1f30-376c-409d-aaeb-da92b67441c2\" (UID: \"f43f1f30-376c-409d-aaeb-da92b67441c2\") "
Apr 17 17:38:03.618286 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.618260    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f43f1f30-376c-409d-aaeb-da92b67441c2" (UID: "f43f1f30-376c-409d-aaeb-da92b67441c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 17 17:38:03.618334 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.618262    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "f43f1f30-376c-409d-aaeb-da92b67441c2" (UID: "f43f1f30-376c-409d-aaeb-da92b67441c2"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 17 17:38:03.620055 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.620033    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f43f1f30-376c-409d-aaeb-da92b67441c2" (UID: "f43f1f30-376c-409d-aaeb-da92b67441c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 17 17:38:03.620174 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.620158    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc" (OuterVolumeSpecName: "kube-api-access-gvdjc") pod "f43f1f30-376c-409d-aaeb-da92b67441c2" (UID: "f43f1f30-376c-409d-aaeb-da92b67441c2"). InnerVolumeSpecName "kube-api-access-gvdjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 17 17:38:03.719013 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.718967    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f1f30-376c-409d-aaeb-da92b67441c2-kserve-provision-location\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:38:03.719013 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.719008    2579 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43f1f30-376c-409d-aaeb-da92b67441c2-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:38:03.719227 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.719024    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43f1f30-376c-409d-aaeb-da92b67441c2-proxy-tls\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:38:03.719227 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:03.719037    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdjc\" (UniqueName: \"kubernetes.io/projected/f43f1f30-376c-409d-aaeb-da92b67441c2-kube-api-access-gvdjc\") on node \"ip-10-0-141-140.ec2.internal\" DevicePath \"\""
Apr 17 17:38:04.340837 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.340806    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/1.log"
Apr 17 17:38:04.341251 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341159    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd_f43f1f30-376c-409d-aaeb-da92b67441c2/storage-initializer/0.log"
Apr 17 17:38:04.341251 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341192    2579 generic.go:358] "Generic (PLEG): container finished" podID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerID="d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2" exitCode=1
Apr 17 17:38:04.341251 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341223    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerDied","Data":"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"}
Apr 17 17:38:04.341359 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341271    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd" event={"ID":"f43f1f30-376c-409d-aaeb-da92b67441c2","Type":"ContainerDied","Data":"3c97e2c5d1fb84d51cfdce5c1faa3e4ae47b88fd0c007b7a0ae7f6df04b04830"}
Apr 17 17:38:04.341359 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341289    2579 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"
Apr 17 17:38:04.341359 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.341292    2579 scope.go:117] "RemoveContainer" containerID="d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"
Apr 17 17:38:04.352863 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.351525    2579 scope.go:117] "RemoveContainer" containerID="4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"
Apr 17 17:38:04.360784 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.360749    2579 scope.go:117] "RemoveContainer" containerID="d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"
Apr 17 17:38:04.361065 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:38:04.361046    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2\": container with ID starting with d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2 not found: ID does not exist" containerID="d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"
Apr 17 17:38:04.361114 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.361074    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2"} err="failed to get container status \"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2\": rpc error: code = NotFound desc = could not find container \"d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2\": container with ID starting with d1ece7b002408c1bf9c90817c2f3d4d3e668b9de881645a213688bdc37b1f0f2 not found: ID does not exist"
Apr 17 17:38:04.361114 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.361092    2579 scope.go:117] "RemoveContainer" containerID="4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"
Apr 17 17:38:04.361338 ip-10-0-141-140 kubenswrapper[2579]: E0417 17:38:04.361319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7\": container with ID starting with 4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7 not found: ID does not exist" containerID="4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"
Apr 17 17:38:04.361386 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.361341    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7"} err="failed to get container status \"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7\": rpc error: code = NotFound desc = could not find container \"4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7\": container with ID starting with 4ca2cf3be97308dc6a152c31b7c247916114bac2d3a0e5f2d5c4ee43c95369e7 not found: ID does not exist"
Apr 17 17:38:04.383923 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.383889    2579 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:38:04.389563 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:04.389530    2579 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-nssmd"]
Apr 17 17:38:05.389611 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:05.389572    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" path="/var/lib/kubelet/pods/f43f1f30-376c-409d-aaeb-da92b67441c2/volumes"
Apr 17 17:38:36.611790 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:36.611739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fgtpp_95e5b703-00aa-4d7f-b79a-e7bd129e8ce7/global-pull-secret-syncer/0.log"
Apr 17 17:38:36.680262 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:36.680220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-67c86_b8ca92f6-eb37-4066-b492-157ff72c4e53/konnectivity-agent/0.log"
Apr 17 17:38:36.872643 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:36.872608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-140.ec2.internal_731cb92bc1983e14e5fcbbe29c5e1e5c/haproxy/0.log"
Apr 17 17:38:40.015882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.015848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/alertmanager/0.log"
Apr 17 17:38:40.062617 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.062581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/config-reloader/0.log"
Apr 17 17:38:40.094720 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.094689    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/kube-rbac-proxy-web/0.log"
Apr 17 17:38:40.124562 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.124532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/kube-rbac-proxy/0.log"
Apr 17 17:38:40.150996 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.150963    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/kube-rbac-proxy-metric/0.log"
Apr 17 17:38:40.177370 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.177339    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/prom-label-proxy/0.log"
Apr 17 17:38:40.204700 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.204669    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e62bcf42-1729-4ba3-82c5-bbdbac97362f/init-config-reloader/0.log"
Apr 17 17:38:40.267251 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.267174    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-75587bd455-mk5cl_dfe89383-3a58-4c17-847d-fe8456a068e8/cluster-monitoring-operator/0.log"
Apr 17 17:38:40.396105 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.396074    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57448c675f-2zlhb_6e7688a4-affd-42fe-8b7d-703f4dba7b49/metrics-server/0.log"
Apr 17 17:38:40.541809 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.541714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5z_463d513e-b523-4a9b-90d8-857aad9bc41a/node-exporter/0.log"
Apr 17 17:38:40.565720 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.565688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5z_463d513e-b523-4a9b-90d8-857aad9bc41a/kube-rbac-proxy/0.log"
Apr 17 17:38:40.590119 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:40.590095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5z_463d513e-b523-4a9b-90d8-857aad9bc41a/init-textfile/0.log"
Apr 17 17:38:41.099976 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:41.099950    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bb84bdc-5k575_089a8be7-7f3d-4d59-922d-88bcd46ee5a4/telemeter-client/0.log"
Apr 17 17:38:41.122929 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:41.122898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bb84bdc-5k575_089a8be7-7f3d-4d59-922d-88bcd46ee5a4/reload/0.log"
Apr 17 17:38:41.147165 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:41.147139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bb84bdc-5k575_089a8be7-7f3d-4d59-922d-88bcd46ee5a4/kube-rbac-proxy/0.log"
Apr 17 17:38:42.913495 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:42.913439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/1.log"
Apr 17 17:38:42.922060 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:42.922028    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-vnhv5_57050c6f-fb02-47e6-96b0-58ebe611a2bb/console-operator/2.log"
Apr 17 17:38:43.284807 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.284701    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8696869745-4qc52_85efab03-3e5a-495a-a123-ef8f18806455/console/0.log"
Apr 17 17:38:43.737868 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.737834    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7c6cbb6c87-sdbzt_98f6bd2e-c563-4fc2-b391-a03bd570bcbc/volume-data-source-validator/0.log"
Apr 17 17:38:43.803717 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.803690    2579 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"]
Apr 17 17:38:43.804034 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804021    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="storage-initializer"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804035    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="storage-initializer"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804045    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerName="storage-initializer"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804050    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerName="storage-initializer"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804062    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804068    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container"
Apr 17 17:38:43.804081 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804079    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kube-rbac-proxy"
Apr 17 17:38:43.804267 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kube-rbac-proxy"
Apr 17 17:38:43.804267 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804145    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kserve-container"
Apr 17 17:38:43.804267 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804154    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="090c7a0e-ff9c-41c0-9150-d49a52f54dcc" containerName="kube-rbac-proxy"
Apr 17 17:38:43.804267 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804160    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerName="storage-initializer"
Apr 17 17:38:43.804267 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.804168    2579 memory_manager.go:356] "RemoveStaleState removing state" podUID="f43f1f30-376c-409d-aaeb-da92b67441c2" containerName="storage-initializer"
Apr 17 17:38:43.807151 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.807135    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.809327 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.809300    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lbcd\"/\"openshift-service-ca.crt\""
Apr 17 17:38:43.809479 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.809359    2579 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lbcd\"/\"kube-root-ca.crt\""
Apr 17 17:38:43.810040 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.810024    2579 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5lbcd\"/\"default-dockercfg-r4zks\""
Apr 17 17:38:43.815330 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.815306    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"]
Apr 17 17:38:43.874355 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.874316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-podres\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.874544 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.874360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-proc\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.874544 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.874392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-sys\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.874544 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.874431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtrm\" (UniqueName: \"kubernetes.io/projected/f479edc5-810b-4dd9-a6fc-3cf8944d764d-kube-api-access-vxtrm\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.874544 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.874512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-lib-modules\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975140 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975110    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-sys\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975165    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtrm\" (UniqueName: \"kubernetes.io/projected/f479edc5-810b-4dd9-a6fc-3cf8944d764d-kube-api-access-vxtrm\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975203    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-lib-modules\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975225    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-podres\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975235    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-sys\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975245    2579 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-proc\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975334    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-podres\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-proc\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.975558 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.975411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f479edc5-810b-4dd9-a6fc-3cf8944d764d-lib-modules\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:43.982629 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:43.982598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtrm\" (UniqueName: \"kubernetes.io/projected/f479edc5-810b-4dd9-a6fc-3cf8944d764d-kube-api-access-vxtrm\") pod \"perf-node-gather-daemonset-7742k\" (UID: \"f479edc5-810b-4dd9-a6fc-3cf8944d764d\") " pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:44.117882 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.117853    2579 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:44.274422 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.274397    2579 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"]
Apr 17 17:38:44.276920 ip-10-0-141-140 kubenswrapper[2579]: W0417 17:38:44.276892    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf479edc5_810b_4dd9_a6fc_3cf8944d764d.slice/crio-f2fb8f1336de9cacf19e1d941d54fc6ef935d7e4b95b864b2f7e78395601cd92 WatchSource:0}: Error finding container f2fb8f1336de9cacf19e1d941d54fc6ef935d7e4b95b864b2f7e78395601cd92: Status 404 returned error can't find the container with id f2fb8f1336de9cacf19e1d941d54fc6ef935d7e4b95b864b2f7e78395601cd92
Apr 17 17:38:44.407587 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.407507    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f8sph_2e7b194d-150a-4fd7-9f3d-e8475c1cd65d/dns/0.log"
Apr 17 17:38:44.429054 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.429030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f8sph_2e7b194d-150a-4fd7-9f3d-e8475c1cd65d/kube-rbac-proxy/0.log"
Apr 17 17:38:44.473796 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.473748    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k" event={"ID":"f479edc5-810b-4dd9-a6fc-3cf8944d764d","Type":"ContainerStarted","Data":"05aafeddfde0a48be0ba2a93fb2171efb5ec33bccdcdaf31c158bf20a0229ac4"}
Apr 17 17:38:44.473796 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.473798    2579 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k" event={"ID":"f479edc5-810b-4dd9-a6fc-3cf8944d764d","Type":"ContainerStarted","Data":"f2fb8f1336de9cacf19e1d941d54fc6ef935d7e4b95b864b2f7e78395601cd92"}
Apr 17 17:38:44.474028 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.473855    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:44.490552 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.490495    2579 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k" podStartSLOduration=1.490477554 podStartE2EDuration="1.490477554s" podCreationTimestamp="2026-04-17 17:38:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-17 17:38:44.489688006 +0000 UTC m=+4051.730346441" watchObservedRunningTime="2026-04-17 17:38:44.490477554 +0000 UTC m=+4051.731135996"
Apr 17 17:38:44.569417 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:44.569389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j5cpk_38c0cc14-3fcb-4193-8823-9a4e0a06f777/dns-node-resolver/0.log"
Apr 17 17:38:45.068748 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:45.068720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76q52_4bd685f1-9eee-4a49-9473-4db1d50bd8b8/node-ca/0.log"
Apr 17 17:38:45.822637 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:45.822606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-574d8f6f84-q4xkf_e0783a18-1157-4c0a-83cf-e1c50ed9d2f7/router/0.log"
Apr 17 17:38:46.154839 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.154809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmlp4_d5c9df82-d83b-4bc7-8eb5-dace7b9eff4a/serve-healthcheck-canary/0.log"
Apr 17 17:38:46.533122 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.533039    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-585dfdc468-k28wq_047abce8-32b2-4b6d-8f0e-2c7ce6ef6401/insights-operator/1.log"
Apr 17 17:38:46.608030 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.608005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-585dfdc468-k28wq_047abce8-32b2-4b6d-8f0e-2c7ce6ef6401/insights-operator/0.log"
Apr 17 17:38:46.782134 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.782108    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbw4n_a3515d44-4ec5-4088-a0d5-d79c4d31314b/kube-rbac-proxy/0.log"
Apr 17 17:38:46.802621 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.802558    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbw4n_a3515d44-4ec5-4088-a0d5-d79c4d31314b/exporter/0.log"
Apr 17 17:38:46.823638 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:46.823616    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dbw4n_a3515d44-4ec5-4088-a0d5-d79c4d31314b/extractor/0.log"
Apr 17 17:38:48.892106 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:48.892077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85bb65f8c4-xr4sz_bdc5ab43-2484-4907-9fe0-3d0db28f6633/manager/0.log"
Apr 17 17:38:49.323199 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:49.323113    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-dzbnm_37533871-b460-45f9-9a46-3e1dbf9b003c/manager/0.log"
Apr 17 17:38:49.467359 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:49.467328    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-j6slw_07e4f192-2403-4ca3-8524-4b565f3a99b7/seaweedfs-tls-serving/0.log"
Apr 17 17:38:50.487670 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:50.487642    2579 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5lbcd/perf-node-gather-daemonset-7742k"
Apr 17 17:38:54.656306 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.656274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/kube-multus-additional-cni-plugins/0.log"
Apr 17 17:38:54.682797 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.682749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/egress-router-binary-copy/0.log"
Apr 17 17:38:54.708025 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.708003    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/cni-plugins/0.log"
Apr 17 17:38:54.728896 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.728876    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/bond-cni-plugin/0.log"
Apr 17 17:38:54.753259 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.753242    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/routeoverride-cni/0.log"
Apr 17 17:38:54.773338 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.773316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/whereabouts-cni-bincopy/0.log"
Apr 17 17:38:54.797227 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:54.797202    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25sc2_0e1bae00-7e41-4dd8-85a3-04bf2d90a6ab/whereabouts-cni/0.log"
Apr 17 17:38:55.195210 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:55.195175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cqjg6_9f980360-c063-4b18-8ccb-25637356228f/kube-multus/0.log"
Apr 17 17:38:55.249936 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:55.249909    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fwn9m_e4163dd4-e68b-4ae1-818d-81e2a6e33a15/network-metrics-daemon/0.log"
Apr 17 17:38:55.268101 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:55.268070    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fwn9m_e4163dd4-e68b-4ae1-818d-81e2a6e33a15/kube-rbac-proxy/0.log"
Apr 17 17:38:56.574725 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.574694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-controller/0.log"
Apr 17 17:38:56.593801 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.593776    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/0.log"
Apr 17 17:38:56.632215 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.632191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovn-acl-logging/1.log"
Apr 17 17:38:56.659072 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.659050    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/kube-rbac-proxy-node/0.log"
Apr 17 17:38:56.681907 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.681883    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/kube-rbac-proxy-ovn-metrics/0.log"
Apr 17 17:38:56.699232 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.699213    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/northd/0.log"
Apr 17 17:38:56.722902 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.722885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/nbdb/0.log"
Apr 17 17:38:56.744804 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.744756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/sbdb/0.log"
Apr 17 17:38:56.932894 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:56.932867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2l4rx_a8f806c8-de16-4bb6-bb39-1871692083fc/ovnkube-controller/0.log"
Apr 17 17:38:58.295141 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:58.295111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-t84cs_e9dc640a-deef-48d0-8af4-5279f7023a59/network-check-target-container/0.log"
Apr 17 17:38:59.163866 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:59.163840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jf79v_17fd98b1-2f78-4fb1-aee8-a9333fd592b4/iptables-alerter/0.log"
Apr 17 17:38:59.883445 ip-10-0-141-140 kubenswrapper[2579]: I0417 17:38:59.883403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bsgct_a0ec9e8b-2f1a-4c9c-96a5-23d9e8deb611/tuned/0.log"