Sep 09 20:52:00.400055 ip-10-0-137-145 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:00.796446 ip-10-0-137-145 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:00.796446 ip-10-0-137-145 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:00.796446 ip-10-0-137-145 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:00.796446 ip-10-0-137-145 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:00.796446 ip-10-0-137-145 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:00.800258 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.800164    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:00.807176 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807153    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.807176 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807172    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.807176 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807178    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807181    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807185    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807188    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807191    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807194    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807197    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807200    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807203    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807205    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807208    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807210    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807213    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807216    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807218    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807221    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807223    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807231    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807234    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807236    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.807276 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807239    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807241    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807244    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807246    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807249    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807252    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807255    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807257    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807262    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807266    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807269    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807272    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807278    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807281    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807284    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807287    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807290    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807293    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807296    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.807765 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807298    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807301    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807303    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807306    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807309    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807311    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807314    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807316    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807319    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807321    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807324    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807328    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807331    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807333    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807337    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807339    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807342    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807345    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807348    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807352    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.808288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807356    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807359    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807362    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807364    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807367    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807370    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807372    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807375    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807378    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807381    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807383    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807386    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807389    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807391    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807394    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807397    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807399    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807404    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807408    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.808804 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807410    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807413    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807416    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807418    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807423    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807425    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807428    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807431    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807434    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807436    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807439    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807442    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807445    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807447    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807450    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807453    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807455    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.807458    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808219    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.809259 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808226    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808229    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808233    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808237    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808240    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808242    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808245    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808249    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808252    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808255    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808257    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808260    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808263    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808266    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808268    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808271    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808274    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808276    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808280    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808282    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.809730 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808285    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808288    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808292    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808295    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808298    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808303    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808306    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808309    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808312    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808315    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808318    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808321    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808323    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808326    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808329    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808331    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808334    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808336    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808339    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.810225 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808342    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808344    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808347    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808349    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808352    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808355    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808358    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808362    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808366    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808368    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808371    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808374    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808377    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808380    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808383    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808387    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808389    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808392    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808395    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808397    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.810718 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808400    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808403    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808406    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808408    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808411    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808413    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808416    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808419    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808423    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808425    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808428    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808430    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808433    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808436    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808438    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808441    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808443    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808446    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808448    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.811212 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808451    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808454    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808457    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808460    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808462    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808466    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808471    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808474    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808477    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808480    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808482    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808485    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808487    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808490    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808492    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808495    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808498    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808501    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808503    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808506    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.811683 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808916    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808922    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808925    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808927    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808930    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808933    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808936    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808939    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808941    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808944    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808947    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808950    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808952    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808955    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808957    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808960    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808963    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808965    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808969    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.812207 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808972    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808975    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808979    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808983    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808985    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808989    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808992    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.808996    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809000    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809002    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809005    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809008    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809011    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809014    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809016    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809019    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809022    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809026    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809028    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.812682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809031    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809034    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809036    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809039    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809042    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809044    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809047    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809049    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809052    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809055    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809057    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809060    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809063    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809066    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809069    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809071    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809074    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809077    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809080    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809083    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.813169 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809085    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809088    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809090    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809098    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809101    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809104    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809106    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809109    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809112    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809115    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809118    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809121    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809123    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809126    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809129    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809132    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809134    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809137    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809139    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809142    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.813717 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809145    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809149    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809152    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809155    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809160    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809163    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809166    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809169    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809172    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809175    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809178    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809181    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809184    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809186    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809189    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809192    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809194    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809197    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809200    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.814288 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809203    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809205    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809278    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809321    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809329    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809335    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809340    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809343    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809347    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809352    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809356    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809360    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809363    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809367    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809370    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809373    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809376    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809380    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809383    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809386    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809390    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809393    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809396    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809399    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:00.814780 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809403    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809406    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809409    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809412    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809416    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809419    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809422    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809425    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809428    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809431    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809435    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809439    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809442    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809445    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809449    2581 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809451    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809456    2581 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809459    2581 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809462    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809465    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809468    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809472    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809475    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809478    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809481    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:00.815390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809484    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809487    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809490    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809493    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809497    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809500    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809503    2581 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809507    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809510    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809513    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809516    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809519    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809522    2581 flags.go:64] FLAG: --help="false"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809525    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-145.ec2.internal"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809528    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809531    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809534    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809538    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809542    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809545    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809548    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809551    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809555    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:00.816004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809558    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809561    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809564    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809567    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809570    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809573    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809576    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809578    2581 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809581    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809585    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809588    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809593    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809596    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809599    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809602    2581 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809605    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809608    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809611    2581 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809614    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809624    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809627    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809632    2581 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809635    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809638    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809641    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:00.816594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809644    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809647    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809652    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809655    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809662    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809665    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809668    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809671    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809674    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809681    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809684    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809687    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809690    2581 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809693    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809708    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08e0d5cba922cf9e6"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809712    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809715    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809718    2581 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809721    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809724    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809728    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809731    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809734    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809738    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809741    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:00.817227 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809745    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809748    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809750    2581 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809753    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809756    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809759    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809762    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809765    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809768    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809773    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809776    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809779    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809782    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809784    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809787    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809790    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809793    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809796    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809802    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809805    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809808    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809812    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809814    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809817    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809822    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:00.818152 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809825    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809829    2581 flags.go:64] FLAG: --v="2"
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809833    2581 flags.go:64] FLAG: --version="false"
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809838    2581 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809841    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.809845    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809965    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809970    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809973    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809977    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809979    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809983    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809987    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809990    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809993    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.809996    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810001    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810004    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810007    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810009    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810012    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.818973 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810015    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810017    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810020    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810022    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810025    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810027    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810030    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810033    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810035    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810038    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810042    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810045    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810047    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810050    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810053    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810056    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810058    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810061    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810064    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810067    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.819503 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810069    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810072    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810074    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810077    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810079    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810082    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810084    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810089    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810092    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810095    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810097    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810100    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810102    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810105    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810108    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810110    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810113    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810115    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810119    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810122    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.820079 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810125    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810128    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810132    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810134    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810137    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810139    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810142    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810144    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810147    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810149    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810152    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810155    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810158    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810160    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810163    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810165    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810168    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810170    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810173    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810176    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.820576 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810179    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810182    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810185    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810187    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810190    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810192    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810195    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810197    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810200    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810202    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810205    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810207    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810210    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810213    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810217    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810220    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810222    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810225    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810228    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.821086 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810232    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810235    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810238    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810241    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.810244    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.810986    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.819923    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.819942    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.820876    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.820884    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820952    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820958    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820962    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820965    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820968    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.821554 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820971    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820974    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820976    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820979    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820982    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820984    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820987    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820990    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820992    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820995    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.820998    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821001    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821004    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821006    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821009    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821011    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821014    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821016    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821021    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.821959 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821024    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821028    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821031    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821034    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821037    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821039    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821042    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821045    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821047    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821050    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821053    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821055    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821058    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821060    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821063    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821065    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821068    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821071    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821073    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821076    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.822456 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821078    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821081    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821084    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821087    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821090    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821093    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821095    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821098    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821101    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821104    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821106    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821109    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821112    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821117    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821121    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821124    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821127    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821129    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821132    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.822949 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821134    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821137    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821139    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821142    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821144    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821147    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821149    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821152    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821156    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821159    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821162    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821165    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821168    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821170    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821173    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821175    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821179    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821182    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821185    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821188    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.823422 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821191    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821193    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821196    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821198    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821201    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821203    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821206    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821209    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821212    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821215    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821217    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821220    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821222    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821225    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821228    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.823906 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821230    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.821236    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821353    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821360    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821363    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821366    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821368    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821371    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821374    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821377    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821379    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821382    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821385    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821388    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821391    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:00.824278 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821394    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821396    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821399    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821401    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821404    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821406    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821409    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821411    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821414    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821417    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821420    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821423    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821427    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821430    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821434    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821437    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821440    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821443    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821446    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:00.824712 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821449    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821452    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821455    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821457    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821460    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821463    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821466    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821468    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821472    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821474    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821477    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821479    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821482    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821485    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821488    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821491    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821493    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821496    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821498    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821501    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:00.825186 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821503    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821506    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821509    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821511    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821515    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821517    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821520    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821523    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821525    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821528    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821531    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821533    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821536    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821539    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821541    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821544    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821547    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821549    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821552    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821554    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:00.825682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821557    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821560    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821562    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821565    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821567    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821570    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821573    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821576    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821578    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821581    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821584    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821586    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821589    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821591    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821594    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821597    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821600    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821603    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821606    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821608    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:00.826183 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821611    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821613    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821616    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821619    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821621    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821624    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:00.821626    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.821632    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.821772    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.824217    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.825328    2581 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.825423    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:00.826682 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.825455    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:00.828212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.828197    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:00.833251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.833233    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:00.842111 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.842092    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:00.845618 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.845599    2581 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:00.846571 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.846537    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:00.849526 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.849498    2581 fs.go:135] Filesystem UUIDs: map[720b9f9e-ca9a-4518-9fb8-4232e7cb1991:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a1f611a1-9d90-4112-9882-2c16b359c93b:/dev/nvme0n1p4]
Sep 09 20:52:00.849603 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.849525    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:00.852946 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.852926    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:00.856245 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.856133    2581 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:00.854749954 +0000 UTC m=+0.351024006 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099234 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29aaf9d93e1b0c5da2e5e739f6a502 SystemUUID:ec29aaf9-d93e-1b0c-5da2-e5e739f6a502 BootID:4e6a0708-88f9-4278-9861-1f59c7726e15 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:37:db:45:2f:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:37:db:45:2f:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:02:dd:87:a4:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:00.856336 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.856245    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:00.856371 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.856336    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:00.859744 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.859715    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:00.859887 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.859749    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:00.860595 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.860585    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:00.860638 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.860598    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:00.860638 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.860606    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:00.861414 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.861403    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:00.863330 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.863318    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:00.868167 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868152    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:00.868288 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868278    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:00.868325 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868310    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:00.868325 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868321    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:00.868393 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868332    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:00.868393 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868342    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:00.868393 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868353    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:00.868754 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868740    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:00.868802 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.868769    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:00.871159 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.871138    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:00.874286 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874269    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:00.874368 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874318    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:00.874578 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874566    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:00.874609 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874585    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:00.874609 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874591    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:00.874609 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874597    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:00.874609 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874605    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:00.874766 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.874611    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:00.875132 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875123    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:00.875160 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875134    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:00.875160 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875142    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:00.875160 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875148    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:00.875160 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875157    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:00.875264 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.875165    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:00.876115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.876097    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:00.876115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.876111    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:00.878303 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.878265    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:00.879091 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.879059    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:00.879091 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.879061    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:00.881080 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.881064    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9hj85"
Sep 09 20:52:00.883323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.883310    2581 server.go:1268] "Started kubelet"
Sep 09 20:52:00.883447 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.883387    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:00.883497 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.883432    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:00.883547 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.883532    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:00.884410 ip-10-0-137-145 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:00.885042 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.885025    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:00.888784 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.888767    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:00.888913 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.888894    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:00.892324 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.891791    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:00.892674 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.892592    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:00.893033 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893008    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9hj85"
Sep 09 20:52:00.893546 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893525    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:00.893546 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893548    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:00.893678 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893631    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:00.893899 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893886    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:00.893899 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.893898    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:00.894010 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.893962    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:00.894567 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.891221    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-145.ec2.internal.18d3c24ed7ff1490  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-145.ec2.internal,UID:ip-10-0-137-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-145.ec2.internal,},FirstTimestamp:2026-09-09 20:52:00.883274896 +0000 UTC m=+0.379548928,LastTimestamp:2026-09-09 20:52:00.883274896 +0000 UTC m=+0.379548928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-145.ec2.internal,}"
Sep 09 20:52:00.895195 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895180    2581 factory.go:153] Registering CRI-O factory
Sep 09 20:52:00.895261 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895228    2581 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:00.895297 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895286    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:00.895341 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895297    2581 factory.go:55] Registering systemd factory
Sep 09 20:52:00.895341 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895306    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:00.895341 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895325    2581 factory.go:103] Registering Raw factory
Sep 09 20:52:00.895468 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.895401    2581 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:00.896752 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.896738    2581 manager.go:349] Starting recovery of all containers
Sep 09 20:52:00.897200 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.897163    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:00.897290 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.897257    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:00.908198 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.908014    2581 manager.go:354] Recovery completed
Sep 09 20:52:00.914383 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.914368    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:00.916377 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.916355    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:00.916481 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.916390    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:00.916481 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.916405    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:00.917128 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.917104    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:00.917128 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.917119    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:00.917274 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.917120    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:00.917274 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.917185    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:00.918649 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.918629    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:00.918783 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.918666    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:00.918783 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.918681    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:00.919428 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.919415    2581 policy_none.go:49] "None policy: Start"
Sep 09 20:52:00.919487 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.919434    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:00.919487 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.919450    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:00.921528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.921515    2581 policy_none.go:47] "Start"
Sep 09 20:52:00.947407 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.947387    2581 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.947435    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.947447    2581 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.947860    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.947873    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.948126    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.948218    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:00.948227    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.948681    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:00.972055 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:00.948747    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.022754 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.022724    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:01.023949 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.023933    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:01.024011 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.023989    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:01.024064 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.024013    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:01.024129 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.024113    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:01.026511 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.026482    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:01.048816 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.048680    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:01.050269 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.050249    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:01.050383 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.050281    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:01.050383 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.050293    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:01.050383 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.050317    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.058103 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.058081    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.058167 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.058111    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-145.ec2.internal\": node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.081257 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.081229    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.124823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.124193    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"]
Sep 09 20:52:01.125004 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.124876    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:01.126551 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.126535    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:01.126626 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.126569    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:01.126626 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.126582    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:01.129055 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.129042    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:01.129180 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.129166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.129215 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.129200    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:01.130288 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130273    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:01.130370 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130299    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:01.130370 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130315    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:01.130370 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130328    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:01.130370 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130351    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:01.130370 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.130364    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:01.132721 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.132690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.132772 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.132739    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:01.133896 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.133880    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:01.133983 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.133913    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:01.133983 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.133927    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:01.154536 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.154509    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-145.ec2.internal\" not found" node="ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.158850 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.158833    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-145.ec2.internal\" not found" node="ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.181330 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.181300    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.194726 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.194685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.194792 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.194732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.194792 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.194750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1001eb5eb924c8ce04785f3e1fa27845-config\") pod \"kube-apiserver-proxy-ip-10-0-137-145.ec2.internal\" (UID: \"1001eb5eb924c8ce04785f3e1fa27845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.282096 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.282066    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.295375 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.295432 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.295432 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1001eb5eb924c8ce04785f3e1fa27845-config\") pod \"kube-apiserver-proxy-ip-10-0-137-145.ec2.internal\" (UID: \"1001eb5eb924c8ce04785f3e1fa27845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.295508 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.295546 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6f78110dc72343914d240e9ce9b26c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal\" (UID: \"ae6f78110dc72343914d240e9ce9b26c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.295546 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.295471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1001eb5eb924c8ce04785f3e1fa27845-config\") pod \"kube-apiserver-proxy-ip-10-0-137-145.ec2.internal\" (UID: \"1001eb5eb924c8ce04785f3e1fa27845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.382592 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.382500    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.460310 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.460266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.461502 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.461482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:01.482634 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.482605    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.589667 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.589605    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.690431 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.690332    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.790798 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.790752    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.826731 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.826684    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:01.827236 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.826930    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:01.890985 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.890955    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:01.896580 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.896549    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:01 +0000 UTC" deadline="2028-04-20 09:56:47.734707788 +0000 UTC"
Sep 09 20:52:01.896580 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:01.896578    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14125h4m45.838133961s"
Sep 09 20:52:01.991776 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:01.991686    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:02.092345 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.092302    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:02.106280 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:02.106235    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1001eb5eb924c8ce04785f3e1fa27845.slice/crio-a55ea995b6af7bac549c67ec3a0d151575ac3f37cfc3f9fefbdf55af02212fa0 WatchSource:0}: Error finding container a55ea995b6af7bac549c67ec3a0d151575ac3f37cfc3f9fefbdf55af02212fa0: Status 404 returned error can't find the container with id a55ea995b6af7bac549c67ec3a0d151575ac3f37cfc3f9fefbdf55af02212fa0
Sep 09 20:52:02.106938 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:02.106916    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f78110dc72343914d240e9ce9b26c.slice/crio-0d529f84ecaca73ca7a8c29b4ec929bf5d3daac2e0bae50a4e4a8366d515bcd9 WatchSource:0}: Error finding container 0d529f84ecaca73ca7a8c29b4ec929bf5d3daac2e0bae50a4e4a8366d515bcd9: Status 404 returned error can't find the container with id 0d529f84ecaca73ca7a8c29b4ec929bf5d3daac2e0bae50a4e4a8366d515bcd9
Sep 09 20:52:02.111429 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.111401    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:02.192413 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.192375    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:02.194038 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.194004    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:02.297880 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.297774    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-145.ec2.internal\" not found"
Sep 09 20:52:02.352712 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.352678    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:02.383930 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.383904    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:02.394993 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.394958    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:02.405546 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.405519    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:02.406374 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.406358    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal"
Sep 09 20:52:02.417249 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.417227    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:02.844919 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.844883    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:02.869510 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.869459    2581 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:02.878398 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.878367    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:02.878826 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.878798    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal","openshift-multus/multus-additional-cni-plugins-vpgql","openshift-multus/network-metrics-daemon-bzt4v","openshift-network-operator/iptables-alerter-7jzx5","openshift-ovn-kubernetes/ovnkube-node-zp8rd","kube-system/konnectivity-agent-bkmnv","kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal","openshift-cluster-node-tuning-operator/tuned-x47rr","openshift-dns/node-resolver-mhfl6","openshift-image-registry/node-ca-ppvcw","openshift-multus/multus-m687m","openshift-network-diagnostics/network-check-target-qn2gw","kube-system/global-pull-secret-syncer-qqzz4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"]
Sep 09 20:52:02.881730 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.881705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:02.884241 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.883946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:02.884241 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.884023    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:02.885740 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.885467    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:02.885740 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.885617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.885740 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.885724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.885977 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.885906    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7shsv\""
Sep 09 20:52:02.886329 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.886302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:02.886422 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.886395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:02.888679 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.888625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:02.894634 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.894602    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.894758 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.894639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:02.894758 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.894669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h8wzh\""
Sep 09 20:52:02.894933 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.894903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.895000 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.894934    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:02.895904 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.895332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.898300 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.898279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:02.898942 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.898543    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w64nt\""
Sep 09 20:52:02.898942 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.898909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.899299 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.899284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:02.899444 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.899374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:02.899604 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.899303    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.899940 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.899920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:02.901176 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.901161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:02.902509 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.902490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:02.904224 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-serviceca\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:02.904300 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:02.904300 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:02.904386 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec54c4a4-6f59-4b21-8e30-b76844beac24-host-slash\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:02.904386 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-bin\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904386 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-node-log\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904548 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904548 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-netd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904548 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdmf\" (UniqueName: \"kubernetes.io/projected/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-kube-api-access-9fdmf\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:02.904548 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:02.904548 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec54c4a4-6f59-4b21-8e30-b76844beac24-iptables-alerter-script\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-etc-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-log-socket\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-config\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcrh\" (UniqueName: \"kubernetes.io/projected/3b650cf8-1258-47a9-a786-3df974f4ae9c-kube-api-access-zbcrh\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.904823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-systemd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-ovn\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-script-lib\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.904979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-host\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9gt\" (UniqueName: \"kubernetes.io/projected/6ab21338-9de8-4bef-a83e-5e42f15a438b-kube-api-access-gs9gt\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4f66\" (UniqueName: \"kubernetes.io/projected/ec54c4a4-6f59-4b21-8e30-b76844beac24-kube-api-access-z4f66\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-systemd-units\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-var-lib-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905143 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-env-overrides\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-dbus\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-netns\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovn-node-metrics-cert\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-kubelet-config\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-kubelet\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-slash\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:02.905667 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f879h\""
Sep 09 20:52:02.906047 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:02.906047 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.905918    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.906269 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.906176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:02.906269 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.906251    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jp6cg\""
Sep 09 20:52:02.907613 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.907530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.907952 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.907916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:02.908584 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.908566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qw2pz\""
Sep 09 20:52:02.910800 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.910752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:02.911037 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.911014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.911602 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.911586    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.911919 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.911892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.911919 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.911911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:02.912978 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.912957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:02.913190 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.913173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rkprc\""
Sep 09 20:52:02.913407 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.913390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:02.914033 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.914016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:02.914316 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.914297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m687m"
Sep 09 20:52:02.915499 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.915468    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:02.916144 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.915882    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:02.916144 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.915958    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5t8x\""
Sep 09 20:52:02.916144 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.915992    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:02.916451 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.916432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:02.916640 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:02.916614    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:02.917123 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.916990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:02.917472 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.917147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9qfbx\""
Sep 09 20:52:02.936014 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.935977    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f2qgw"
Sep 09 20:52:02.942559 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.942529    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f2qgw"
Sep 09 20:52:02.994745 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:02.994714    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:03.005831 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.005794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9gt\" (UniqueName: \"kubernetes.io/projected/6ab21338-9de8-4bef-a83e-5e42f15a438b-kube-api-access-gs9gt\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:03.006030 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.005842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f66\" (UniqueName: \"kubernetes.io/projected/ec54c4a4-6f59-4b21-8e30-b76844beac24-kube-api-access-z4f66\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.006092 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-sys-fs\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.006153 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.006153 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx95b\" (UniqueName: \"kubernetes.io/projected/287066b7-5bb3-4553-9818-ffd2b8840bf9-kube-api-access-kx95b\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.006270 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:03.006270 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.006270 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-env-overrides\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.006270 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-dbus\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-k8s-cni-cncf-io\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-netns\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-kubelet\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovn-node-metrics-cert\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-kubelet-config\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-konnectivity-ca\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.006528 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-host\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-kubelet\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bb328a0-4ea0-4e63-9965-a10f79458da8-hosts-file\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-kubelet\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysconfig\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-kubelet-config\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2gc\" (UniqueName: \"kubernetes.io/projected/04788cd2-aff7-4451-9ff0-2525e1659638-kube-api-access-pg2gc\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-dbus\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-run\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-lib-modules\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-tmp\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.006928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-hostroot\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-node-log\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-socket-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-device-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-system-cni-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-conf-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-conf\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-systemd\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-sys\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-env-overrides\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-socket-dir-parent\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-multus-certs\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-etc-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-node-log\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.007541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-config\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcrh\" (UniqueName: \"kubernetes.io/projected/3b650cf8-1258-47a9-a786-3df974f4ae9c-kube-api-access-zbcrh\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-agent-certs\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007873    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-etc-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.007958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.007987    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-etc-tuned\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.008077    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:03.508042978 +0000 UTC m=+3.004317001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-systemd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-script-lib\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-systemd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb328a0-4ea0-4e63-9965-a10f79458da8-tmp-dir\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-systemd-units\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-var-lib-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvr9h\" (UniqueName: \"kubernetes.io/projected/e0cfceed-bd31-4875-86ee-8282eec94e9d-kube-api-access-nvr9h\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-var-lib-openvswitch\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-modprobe-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-systemd-units\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-var-lib-kubelet\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-multus\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkn8c\" (UniqueName: \"kubernetes.io/projected/1b25be79-7378-407a-88e9-068d191bd16d-kube-api-access-hkn8c\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-netns\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-cnibin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-multus-daemon-config\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-slash\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-run-netns\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-serviceca\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.008932 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-config\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec54c4a4-6f59-4b21-8e30-b76844beac24-host-slash\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-bin\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-cnibin\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.008858    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec54c4a4-6f59-4b21-8e30-b76844beac24-host-slash\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-cni-binary-copy\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fk5t\" (UniqueName: \"kubernetes.io/projected/7bb328a0-4ea0-4e63-9965-a10f79458da8-kube-api-access-5fk5t\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.008919    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:03.508901399 +0000 UTC m=+3.005175434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-slash\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovnkube-script-lib\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-kubernetes\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.008968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-bin\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-netd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdmf\" (UniqueName: \"kubernetes.io/projected/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-kube-api-access-9fdmf\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec54c4a4-6f59-4b21-8e30-b76844beac24-iptables-alerter-script\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.009785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-registration-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-log-socket\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-serviceca\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-log-socket\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-host-cni-netd\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-system-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-bin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-etc-kubernetes\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-ovn\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-host\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b650cf8-1258-47a9-a786-3df974f4ae9c-run-ovn\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-host\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.010418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.009647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec54c4a4-6f59-4b21-8e30-b76844beac24-iptables-alerter-script\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.013032 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.013004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b650cf8-1258-47a9-a786-3df974f4ae9c-ovn-node-metrics-cert\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.019448 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.019420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9gt\" (UniqueName: \"kubernetes.io/projected/6ab21338-9de8-4bef-a83e-5e42f15a438b-kube-api-access-gs9gt\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:03.020837 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.020569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f66\" (UniqueName: \"kubernetes.io/projected/ec54c4a4-6f59-4b21-8e30-b76844beac24-kube-api-access-z4f66\") pod \"iptables-alerter-7jzx5\" (UID: \"ec54c4a4-6f59-4b21-8e30-b76844beac24\") " pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.020837 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.020746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcrh\" (UniqueName: \"kubernetes.io/projected/3b650cf8-1258-47a9-a786-3df974f4ae9c-kube-api-access-zbcrh\") pod \"ovnkube-node-zp8rd\" (UID: \"3b650cf8-1258-47a9-a786-3df974f4ae9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.020837 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.020754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdmf\" (UniqueName: \"kubernetes.io/projected/75d33de8-9ddb-4f97-9691-1ee5220ba6cb-kube-api-access-9fdmf\") pod \"node-ca-ppvcw\" (UID: \"75d33de8-9ddb-4f97-9691-1ee5220ba6cb\") " pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.030028 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.029973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal" event={"ID":"ae6f78110dc72343914d240e9ce9b26c","Type":"ContainerStarted","Data":"0d529f84ecaca73ca7a8c29b4ec929bf5d3daac2e0bae50a4e4a8366d515bcd9"}
Sep 09 20:52:03.031519 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.031493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal" event={"ID":"1001eb5eb924c8ce04785f3e1fa27845","Type":"ContainerStarted","Data":"a55ea995b6af7bac549c67ec3a0d151575ac3f37cfc3f9fefbdf55af02212fa0"}
Sep 09 20:52:03.109989 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.109890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2gc\" (UniqueName: \"kubernetes.io/projected/04788cd2-aff7-4451-9ff0-2525e1659638-kube-api-access-pg2gc\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.109989 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.109941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-run\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.109989 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.109968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-lib-modules\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.109993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-tmp\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-hostroot\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-socket-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-device-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-system-cni-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-conf-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-conf\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110206 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-systemd\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-sys\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-socket-dir-parent\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-multus-certs\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-agent-certs\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-etc-tuned\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-run\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb328a0-4ea0-4e63-9965-a10f79458da8-tmp-dir\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.110785 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-socket-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.110830 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-hostroot\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110866 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb328a0-4ea0-4e63-9965-a10f79458da8-tmp-dir\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.110906 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-multus-certs\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.110906 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-systemd\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110906 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-sys\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.110906 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-conf-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.111028 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.110925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-socket-dir-parent\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.111063 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-conf\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.111095 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-system-cni-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvr9h\" (UniqueName: \"kubernetes.io/projected/e0cfceed-bd31-4875-86ee-8282eec94e9d-kube-api-access-nvr9h\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-modprobe-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-modprobe-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-var-lib-kubelet\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-multus\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkn8c\" (UniqueName: \"kubernetes.io/projected/1b25be79-7378-407a-88e9-068d191bd16d-kube-api-access-hkn8c\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-cnibin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-multus-daemon-config\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-cnibin\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-cni-binary-copy\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-lib-modules\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fk5t\" (UniqueName: \"kubernetes.io/projected/7bb328a0-4ea0-4e63-9965-a10f79458da8-kube-api-access-5fk5t\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-kubernetes\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-registration-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-device-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.113267 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-system-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-bin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-etc-kubernetes\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-sys-fs\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx95b\" (UniqueName: \"kubernetes.io/projected/287066b7-5bb3-4553-9818-ffd2b8840bf9-kube-api-access-kx95b\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-k8s-cni-cncf-io\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-netns\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-kubelet\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-konnectivity-ca\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.111991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-host\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.114323 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bb328a0-4ea0-4e63-9965-a10f79458da8-hosts-file\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysconfig\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-multus\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-multus-daemon-config\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysconfig\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-var-lib-kubelet\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-konnectivity-ca\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-cnibin\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-multus-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-kubernetes\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-registration-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-cnibin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-etc-sysctl-d\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04788cd2-aff7-4451-9ff0-2525e1659638-host\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-system-cni-dir\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-cni-bin\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.115212 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-etc-kubernetes\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.112890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-sys-fs\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/287066b7-5bb3-4553-9818-ffd2b8840bf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bb328a0-4ea0-4e63-9965-a10f79458da8-hosts-file\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-var-lib-kubelet\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-k8s-cni-cncf-io\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0cfceed-bd31-4875-86ee-8282eec94e9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b25be79-7378-407a-88e9-068d191bd16d-host-run-netns\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/287066b7-5bb3-4553-9818-ffd2b8840bf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.113691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-tmp\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.114324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b25be79-7378-407a-88e9-068d191bd16d-cni-binary-copy\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.114365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a8162d3-7b99-4095-8d27-0aa11f28eb5c-agent-certs\") pod \"konnectivity-agent-bkmnv\" (UID: \"1a8162d3-7b99-4095-8d27-0aa11f28eb5c\") " pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.116115 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.115794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04788cd2-aff7-4451-9ff0-2525e1659638-etc-tuned\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.123413 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.123347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2gc\" (UniqueName: \"kubernetes.io/projected/04788cd2-aff7-4451-9ff0-2525e1659638-kube-api-access-pg2gc\") pod \"tuned-x47rr\" (UID: \"04788cd2-aff7-4451-9ff0-2525e1659638\") " pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.125501 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.125309    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:03.125501 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.125337    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:03.125501 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.125354    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:03.125501 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.125432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:03.625410976 +0000 UTC m=+3.121685014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:03.125824 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.125588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx95b\" (UniqueName: \"kubernetes.io/projected/287066b7-5bb3-4553-9818-ffd2b8840bf9-kube-api-access-kx95b\") pod \"multus-additional-cni-plugins-vpgql\" (UID: \"287066b7-5bb3-4553-9818-ffd2b8840bf9\") " pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.125973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.125944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvr9h\" (UniqueName: \"kubernetes.io/projected/e0cfceed-bd31-4875-86ee-8282eec94e9d-kube-api-access-nvr9h\") pod \"aws-ebs-csi-driver-node-jjv2g\" (UID: \"e0cfceed-bd31-4875-86ee-8282eec94e9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.126382 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.126300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fk5t\" (UniqueName: \"kubernetes.io/projected/7bb328a0-4ea0-4e63-9965-a10f79458da8-kube-api-access-5fk5t\") pod \"node-resolver-mhfl6\" (UID: \"7bb328a0-4ea0-4e63-9965-a10f79458da8\") " pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.128171 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.128148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkn8c\" (UniqueName: \"kubernetes.io/projected/1b25be79-7378-407a-88e9-068d191bd16d-kube-api-access-hkn8c\") pod \"multus-m687m\" (UID: \"1b25be79-7378-407a-88e9-068d191bd16d\") " pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.202645 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.202608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ppvcw"
Sep 09 20:52:03.212981 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.212948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jzx5"
Sep 09 20:52:03.224992 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.224962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:03.232169 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.232138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:03.244340 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.244310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhfl6"
Sep 09 20:52:03.252526 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.252500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g"
Sep 09 20:52:03.259623 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.259593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vpgql"
Sep 09 20:52:03.267885 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.267858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x47rr"
Sep 09 20:52:03.273878 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.273857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m687m"
Sep 09 20:52:03.516218 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.516143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:03.516218 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.516193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:03.516415 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.516308    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:03.516415 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.516378    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:04.516362684 +0000 UTC m=+4.012636721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:03.516415 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.516308    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:03.516536 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.516445    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:04.516432106 +0000 UTC m=+4.012706127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:03.718130 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.718078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:03.718300 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.718267    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:03.718352 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.718300    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:03.718352 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.718315    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:03.718425 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:03.718383    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:04.718363678 +0000 UTC m=+4.214637713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:03.946035 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.945911    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:03 +0000 UTC" deadline="2028-06-17 07:12:55.30950921 +0000 UTC"
Sep 09 20:52:03.946035 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:03.945985    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15514h20m51.363529279s"
Sep 09 20:52:04.214951 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.214731    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb328a0_4ea0_4e63_9965_a10f79458da8.slice/crio-24bb3994c1efb5d3c5714b1002ba749ee284fd9a0d0e233dd1ed966b2741b942 WatchSource:0}: Error finding container 24bb3994c1efb5d3c5714b1002ba749ee284fd9a0d0e233dd1ed966b2741b942: Status 404 returned error can't find the container with id 24bb3994c1efb5d3c5714b1002ba749ee284fd9a0d0e233dd1ed966b2741b942
Sep 09 20:52:04.215872 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.215847    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b650cf8_1258_47a9_a786_3df974f4ae9c.slice/crio-cd60681b2f9955549da959abe9b5d363d2a2a6a4d7dc2008c9c1b6b251493c39 WatchSource:0}: Error finding container cd60681b2f9955549da959abe9b5d363d2a2a6a4d7dc2008c9c1b6b251493c39: Status 404 returned error can't find the container with id cd60681b2f9955549da959abe9b5d363d2a2a6a4d7dc2008c9c1b6b251493c39
Sep 09 20:52:04.216656 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.216626    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8162d3_7b99_4095_8d27_0aa11f28eb5c.slice/crio-8dc9fd7cbf12aaade9a6defe9347b49a3575c15ed6ba7383453bf63b7c283f7c WatchSource:0}: Error finding container 8dc9fd7cbf12aaade9a6defe9347b49a3575c15ed6ba7383453bf63b7c283f7c: Status 404 returned error can't find the container with id 8dc9fd7cbf12aaade9a6defe9347b49a3575c15ed6ba7383453bf63b7c283f7c
Sep 09 20:52:04.217637 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.217600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d33de8_9ddb_4f97_9691_1ee5220ba6cb.slice/crio-e5992ae4317379dada2b6ed42c4e52e49cf1290169be86adfe2b4db4f09c4943 WatchSource:0}: Error finding container e5992ae4317379dada2b6ed42c4e52e49cf1290169be86adfe2b4db4f09c4943: Status 404 returned error can't find the container with id e5992ae4317379dada2b6ed42c4e52e49cf1290169be86adfe2b4db4f09c4943
Sep 09 20:52:04.218477 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.218450    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b25be79_7378_407a_88e9_068d191bd16d.slice/crio-ae14321cd045232cc8fb349a5023b503cffc933720e92ad335d6b0a3090dd718 WatchSource:0}: Error finding container ae14321cd045232cc8fb349a5023b503cffc933720e92ad335d6b0a3090dd718: Status 404 returned error can't find the container with id ae14321cd045232cc8fb349a5023b503cffc933720e92ad335d6b0a3090dd718
Sep 09 20:52:04.223521 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.223311    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec54c4a4_6f59_4b21_8e30_b76844beac24.slice/crio-037dfa544085aff2ee90a3047745f0081927a1e4d6e5cbf870f6c38d48f2479e WatchSource:0}: Error finding container 037dfa544085aff2ee90a3047745f0081927a1e4d6e5cbf870f6c38d48f2479e: Status 404 returned error can't find the container with id 037dfa544085aff2ee90a3047745f0081927a1e4d6e5cbf870f6c38d48f2479e
Sep 09 20:52:04.223998 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.223973    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287066b7_5bb3_4553_9818_ffd2b8840bf9.slice/crio-1f722118fe3d7240b5924e7d4dd1e745e05855c796dd35ef765ae854d0e5d207 WatchSource:0}: Error finding container 1f722118fe3d7240b5924e7d4dd1e745e05855c796dd35ef765ae854d0e5d207: Status 404 returned error can't find the container with id 1f722118fe3d7240b5924e7d4dd1e745e05855c796dd35ef765ae854d0e5d207
Sep 09 20:52:04.224937 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.224890    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cfceed_bd31_4875_86ee_8282eec94e9d.slice/crio-3a94497a407065c2c878ee433152a616f54d2f1922be1d47e80fbf1e54e106de WatchSource:0}: Error finding container 3a94497a407065c2c878ee433152a616f54d2f1922be1d47e80fbf1e54e106de: Status 404 returned error can't find the container with id 3a94497a407065c2c878ee433152a616f54d2f1922be1d47e80fbf1e54e106de
Sep 09 20:52:04.227290 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:04.226468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04788cd2_aff7_4451_9ff0_2525e1659638.slice/crio-d4ff68a1b99bdcfc70041a2e45e5c7c07b8d1f5b92a8b5b3dee5214f112412b0 WatchSource:0}: Error finding container d4ff68a1b99bdcfc70041a2e45e5c7c07b8d1f5b92a8b5b3dee5214f112412b0: Status 404 returned error can't find the container with id d4ff68a1b99bdcfc70041a2e45e5c7c07b8d1f5b92a8b5b3dee5214f112412b0
Sep 09 20:52:04.525306 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:04.525277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:04.525426 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:04.525332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:04.525500 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.525458    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:04.525500 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.525473    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:04.525605 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.525537    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:06.52551634 +0000 UTC m=+6.021790377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:04.525605 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.525555    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:06.52554663 +0000 UTC m=+6.021820666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:04.728107 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:04.728010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:04.728285 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.728214    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:04.728285 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.728241    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:04.728285 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.728255    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:04.728440 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:04.728336    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:06.728312953 +0000 UTC m=+6.224586997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:04.946940 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:04.946890    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:03 +0000 UTC" deadline="2028-03-28 23:26:21.127226959 +0000 UTC"
Sep 09 20:52:04.946940 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:04.946939    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13586h34m16.180293895s"
Sep 09 20:52:05.025444 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.024617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:05.025444 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:05.024773    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:05.025444 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.025225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:05.025444 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:05.025328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:05.026014 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.025864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:05.026014 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:05.025978    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:05.036497 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.036422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g" event={"ID":"e0cfceed-bd31-4875-86ee-8282eec94e9d","Type":"ContainerStarted","Data":"3a94497a407065c2c878ee433152a616f54d2f1922be1d47e80fbf1e54e106de"}
Sep 09 20:52:05.039100 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.039070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ppvcw" event={"ID":"75d33de8-9ddb-4f97-9691-1ee5220ba6cb","Type":"ContainerStarted","Data":"e5992ae4317379dada2b6ed42c4e52e49cf1290169be86adfe2b4db4f09c4943"}
Sep 09 20:52:05.040882 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.040853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m687m" event={"ID":"1b25be79-7378-407a-88e9-068d191bd16d","Type":"ContainerStarted","Data":"ae14321cd045232cc8fb349a5023b503cffc933720e92ad335d6b0a3090dd718"}
Sep 09 20:52:05.042876 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.042849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"cd60681b2f9955549da959abe9b5d363d2a2a6a4d7dc2008c9c1b6b251493c39"}
Sep 09 20:52:05.045456 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.045431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhfl6" event={"ID":"7bb328a0-4ea0-4e63-9965-a10f79458da8","Type":"ContainerStarted","Data":"24bb3994c1efb5d3c5714b1002ba749ee284fd9a0d0e233dd1ed966b2741b942"}
Sep 09 20:52:05.048439 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.048410    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae6f78110dc72343914d240e9ce9b26c" containerID="109fece80ad36b88600775cfa863bf0321af0129b9f488597abcc4fc1f077625" exitCode=0
Sep 09 20:52:05.049054 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.049031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal" event={"ID":"ae6f78110dc72343914d240e9ce9b26c","Type":"ContainerDied","Data":"109fece80ad36b88600775cfa863bf0321af0129b9f488597abcc4fc1f077625"}
Sep 09 20:52:05.051174 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.051147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerStarted","Data":"1f722118fe3d7240b5924e7d4dd1e745e05855c796dd35ef765ae854d0e5d207"}
Sep 09 20:52:05.057319 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.057285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jzx5" event={"ID":"ec54c4a4-6f59-4b21-8e30-b76844beac24","Type":"ContainerStarted","Data":"037dfa544085aff2ee90a3047745f0081927a1e4d6e5cbf870f6c38d48f2479e"}
Sep 09 20:52:05.060244 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.060217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bkmnv" event={"ID":"1a8162d3-7b99-4095-8d27-0aa11f28eb5c","Type":"ContainerStarted","Data":"8dc9fd7cbf12aaade9a6defe9347b49a3575c15ed6ba7383453bf63b7c283f7c"}
Sep 09 20:52:05.063478 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.063384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal" event={"ID":"1001eb5eb924c8ce04785f3e1fa27845","Type":"ContainerStarted","Data":"e4c5a2bc0cf9f397142236d93a2a513510d3aee0d140e2cbc444f84691807f35"}
Sep 09 20:52:05.065056 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:05.064958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x47rr" event={"ID":"04788cd2-aff7-4451-9ff0-2525e1659638","Type":"ContainerStarted","Data":"d4ff68a1b99bdcfc70041a2e45e5c7c07b8d1f5b92a8b5b3dee5214f112412b0"}
Sep 09 20:52:06.078897 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:06.078758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal" event={"ID":"ae6f78110dc72343914d240e9ce9b26c","Type":"ContainerStarted","Data":"d0db4cdd07270a93ea83f355c1629c0d7daa7a82451db97f345ac363d968a70b"}
Sep 09 20:52:06.096188 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:06.095331    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-145.ec2.internal" podStartSLOduration=3.095308525 podStartE2EDuration="3.095308525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:05.093416823 +0000 UTC m=+4.589690884" watchObservedRunningTime="2026-09-09 20:52:06.095308525 +0000 UTC m=+5.591582589"
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:06.544843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:06.544917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.545081    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.545144    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.545122774 +0000 UTC m=+10.041396800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.545564    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:06.545719 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.545619    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.545601394 +0000 UTC m=+10.041875434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:06.746276 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:06.746240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:06.746479 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.746412    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:06.746479 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.746436    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:06.746479 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.746450    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:06.746652 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:06.746555    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.74649172 +0000 UTC m=+10.242765749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:07.027608 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:07.027519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:07.027869 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:07.027668    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:07.028193 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:07.028144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:07.028390 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:07.028351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:07.028491 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:07.028445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:07.028555 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:07.028527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:09.025599    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:09.025763    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:09.026186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:09.026285    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:09.026338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:09.026476 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:09.026407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:10.583654 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:10.583603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:10.584177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:10.583722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:10.584177 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.583764    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:10.584177 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.583857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:18.583832616 +0000 UTC m=+18.080106654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:10.584177 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.583888    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:10.584177 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.583956    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:18.583936442 +0000 UTC m=+18.080210480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:10.785566 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:10.785524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:10.785784 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.785729    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:10.785784 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.785766    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:10.785784 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.785778    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:10.785973 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:10.785842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:18.785823046 +0000 UTC m=+18.282097084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:11.025773 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:11.025124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:11.025773 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:11.025260    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:11.025773 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:11.025608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:11.027035 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:11.026833    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:11.027035 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:11.026914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:11.027035 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:11.026998    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:13.024343 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:13.024304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:13.024847 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:13.024372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:13.024847 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:13.024385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:13.024847 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:13.024466    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:13.024847 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:13.024556    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:13.024847 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:13.024661    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:15.025181 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:15.025144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:15.025648 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:15.025144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:15.025648 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:15.025286    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:15.025648 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:15.025328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:15.025648 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:15.025161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:15.025648 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:15.025405    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:17.024979 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:17.024764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:17.025409 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:17.024764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:17.025409 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:17.025068    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:17.025409 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:17.025163    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:17.025409 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:17.024764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:17.025409 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:17.025277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:18.651687 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:18.651644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:18.651687 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:18.651709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:18.652176 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.651807    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:18.652176 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.651808    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:18.652176 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.651859    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:34.651845552 +0000 UTC m=+34.148119586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:18.652176 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.651873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:34.651866627 +0000 UTC m=+34.148140647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:18.853263 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:18.853221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:18.853442 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.853402    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:18.853442 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.853425    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:18.853442 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.853435    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:18.853549 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:18.853493    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:34.853474706 +0000 UTC m=+34.349748730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:19.024627 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:19.024444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:19.024627 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:19.024475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:19.024627 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:19.024457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:19.024627 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:19.024586    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:19.024996 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:19.024667    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:19.024996 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:19.024764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:21.025084 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:21.025046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:21.025557 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:21.025150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:21.025557 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:21.025198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:21.025557 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:21.025320    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:21.025557 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:21.025334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:21.025557 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:21.025416    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:23.024745 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.024554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:23.025251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.024590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:23.025251 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:23.024825    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:23.025251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.024595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:23.025251 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:23.024921    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:23.025251 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:23.024971    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:23.115119 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.115087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x47rr" event={"ID":"04788cd2-aff7-4451-9ff0-2525e1659638","Type":"ContainerStarted","Data":"5c8df4a38471e168aac9125e4ff6776e1b5e1c953facbd2be1d695909d2a5421"}
Sep 09 20:52:23.116552 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.116525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g" event={"ID":"e0cfceed-bd31-4875-86ee-8282eec94e9d","Type":"ContainerStarted","Data":"3ea980b415573d6e7d6f44cf0dc8fb32840d97d9aa6518b1b7db2812400689c8"}
Sep 09 20:52:23.117918 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.117894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ppvcw" event={"ID":"75d33de8-9ddb-4f97-9691-1ee5220ba6cb","Type":"ContainerStarted","Data":"e4ae1b7d11997fa93ce961f92905cd616c40ceeb309dde466068c9749df0b9b3"}
Sep 09 20:52:23.119376 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.119354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m687m" event={"ID":"1b25be79-7378-407a-88e9-068d191bd16d","Type":"ContainerStarted","Data":"3f1022fe909e10a94715df2441c14c83039286f94b10a51699a373cf5ebb9d69"}
Sep 09 20:52:23.122350 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.122323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"39e26e51ce835a741fdf5c74e0dc6bd49d1dc823958b398437035be68e91e9af"}
Sep 09 20:52:23.122441 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.122384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"ef8ed36dec55fcbd4aaf7a68ea7a6e81bdcf931c3518b73bdd201837891c56f8"}
Sep 09 20:52:23.122441 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.122424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"92f578ac2928585ccc93b489521d02c003955c0d19f9c226be9f89a82d5f048f"}
Sep 09 20:52:23.122539 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.122467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"ddfa96c1b85848e5bd66370251c4f0d7a7b21c751e15ae16c02a475bb989a45f"}
Sep 09 20:52:23.122539 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.122498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"c094bc04f5081690a7dd95c5af5dae89f2caeb3c39368bd526686e65f7c65a07"}
Sep 09 20:52:23.126596 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.126542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhfl6" event={"ID":"7bb328a0-4ea0-4e63-9965-a10f79458da8","Type":"ContainerStarted","Data":"09dd26be34f82c6e0182463e3fd1815c89e952478df046534e9de060426cfaf8"}
Sep 09 20:52:23.128610 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.128582    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="ab69e890d02740cdf9ab512767f6a778fa68d95b9b082df1a827ed9c0976cd89" exitCode=0
Sep 09 20:52:23.128727 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.128630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"ab69e890d02740cdf9ab512767f6a778fa68d95b9b082df1a827ed9c0976cd89"}
Sep 09 20:52:23.130224 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.130191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bkmnv" event={"ID":"1a8162d3-7b99-4095-8d27-0aa11f28eb5c","Type":"ContainerStarted","Data":"8bc9feecd7c2c3d553cc9197cfc7dce1872b281d128290b0f7e6e7c5bffd6994"}
Sep 09 20:52:23.136165 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.136129    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-145.ec2.internal" podStartSLOduration=20.136116945 podStartE2EDuration="20.136116945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:06.096151163 +0000 UTC m=+5.592425210" watchObservedRunningTime="2026-09-09 20:52:23.136116945 +0000 UTC m=+22.632390986"
Sep 09 20:52:23.136516 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.136494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x47rr" podStartSLOduration=2.761240965 podStartE2EDuration="21.136484683s" totalImagesPullingTime="18.375243718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:23.136115799 +0000 UTC m=+22.632389841" watchObservedRunningTime="2026-09-09 20:52:23.136484683 +0000 UTC m=+22.632758721"
Sep 09 20:52:23.154156 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.154114    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mhfl6" podStartSLOduration=2.766407841 podStartE2EDuration="21.15409904s" totalImagesPullingTime="18.387691199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:23.154053827 +0000 UTC m=+22.650327889" watchObservedRunningTime="2026-09-09 20:52:23.15409904 +0000 UTC m=+22.650373083"
Sep 09 20:52:23.220501 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.220446    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m687m" podStartSLOduration=2.772711573 podStartE2EDuration="21.220430844s" totalImagesPullingTime="18.447719271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:23.219814492 +0000 UTC m=+22.716088535" watchObservedRunningTime="2026-09-09 20:52:23.220430844 +0000 UTC m=+22.716704882"
Sep 09 20:52:23.237254 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.237207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ppvcw" podStartSLOduration=2.854758861 podStartE2EDuration="21.237190748s" totalImagesPullingTime="18.382431887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:23.236740631 +0000 UTC m=+22.733014682" watchObservedRunningTime="2026-09-09 20:52:23.237190748 +0000 UTC m=+22.733464789"
Sep 09 20:52:23.258898 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.258842    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bkmnv" podStartSLOduration=2.876551952 podStartE2EDuration="21.258826226s" totalImagesPullingTime="18.382274274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:23.258447415 +0000 UTC m=+22.754721456" watchObservedRunningTime="2026-09-09 20:52:23.258826226 +0000 UTC m=+22.755100267"
Sep 09 20:52:23.313395 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.313363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:23.314053 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:23.314037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:24.136016 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.135982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"e10ffa0d4b97b8b4dbb72d74b64ee5e1f98d085e0d247157e4a3a7c57e5ee00e"}
Sep 09 20:52:24.137389 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.137360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jzx5" event={"ID":"ec54c4a4-6f59-4b21-8e30-b76844beac24","Type":"ContainerStarted","Data":"899367ba12bedf1fccd05479a866683cf13e9a1632319272b28f94ee794f3cb3"}
Sep 09 20:52:24.138024 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.137994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:24.138578 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.138563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bkmnv"
Sep 09 20:52:24.183291 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.183248    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7jzx5" podStartSLOduration=3.804656014 podStartE2EDuration="22.183232438s" totalImagesPullingTime="18.378576424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:24.155485467 +0000 UTC m=+23.651759510" watchObservedRunningTime="2026-09-09 20:52:24.183232438 +0000 UTC m=+23.679506480"
Sep 09 20:52:24.482934 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.482742    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:52:24.963554 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.963445    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:24.482928891Z","UUID":"a0d1ac57-f1a9-4fa9-af0a-f927c79a6b61","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:52:24.964832 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.964817    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:52:24.964832 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:24.964843    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:52:25.025187 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.025149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:25.025334 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.025151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:25.025334 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.025306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:25.025334 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:25.025300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:25.025474 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:25.025424    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:25.025514 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:25.025483    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:25.142973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.142877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g" event={"ID":"e0cfceed-bd31-4875-86ee-8282eec94e9d","Type":"ContainerStarted","Data":"24303cb97fb333f5e69c52a0fad6c5b9a091da79313b2408f9e21428b260b55b"}
Sep 09 20:52:25.534172 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.534125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:25.633094 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:25.633056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:26.147661 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:26.147628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"c31032dc1777eb06c617e8f8377608d0e539c14e0c3b4ad675fd2fb2df00707b"}
Sep 09 20:52:26.149676 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:26.149646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g" event={"ID":"e0cfceed-bd31-4875-86ee-8282eec94e9d","Type":"ContainerStarted","Data":"e2cde96fbc35622bb179dd9f5f57e606ad40c016aa5995e8e6c4f7686a04a42c"}
Sep 09 20:52:27.024671 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:27.024644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:27.024671 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:27.024667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:27.024941 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:27.024647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:27.024941 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:27.024764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:27.024941 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:27.024856    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:27.024941 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:27.024921    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:29.024662 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.024620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:29.024974 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.024621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:29.024974 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.024646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:29.024974 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:29.024773    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:29.024974 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:29.024829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:29.024974 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:29.024934    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:29.160350 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.160180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" event={"ID":"3b650cf8-1258-47a9-a786-3df974f4ae9c","Type":"ContainerStarted","Data":"dd714b81c566deea39797009683e9cc5868ef2ee45053e54a4cfa4254adf2533"}
Sep 09 20:52:29.160552 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.160397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:29.160552 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.160439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:29.176795 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.176685    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:29.176967 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.176952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:29.190228 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.190167    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjv2g" podStartSLOduration=6.305678961 podStartE2EDuration="27.190146491s" totalImagesPullingTime="20.88446753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:26.171005447 +0000 UTC m=+25.667279509" watchObservedRunningTime="2026-09-09 20:52:29.190146491 +0000 UTC m=+28.686420535"
Sep 09 20:52:29.190541 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:29.190509    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd" podStartSLOduration=8.749199493 podStartE2EDuration="27.190496653s" totalImagesPullingTime="18.44129716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:29.18937329 +0000 UTC m=+28.685647356" watchObservedRunningTime="2026-09-09 20:52:29.190496653 +0000 UTC m=+28.686770697"
Sep 09 20:52:30.164675 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.164639    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="f4ee7205eb5f612aa4f79657cc4cd03b1c5dad669632b0522aa3c74823a60ed3" exitCode=0
Sep 09 20:52:30.165339 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.164727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"f4ee7205eb5f612aa4f79657cc4cd03b1c5dad669632b0522aa3c74823a60ed3"}
Sep 09 20:52:30.165339 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.164808    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:52:30.321210 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.321131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:30.383565 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.383527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:30.754418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.754380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:30.840854 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.840609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:30.927122 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.927093    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzt4v"]
Sep 09 20:52:30.927331 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.927244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:30.927393 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:30.927344    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:30.929995 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.929970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qn2gw"]
Sep 09 20:52:30.930120 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.930074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:30.930180 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:30.930150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:30.930796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.930773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqzz4"]
Sep 09 20:52:30.930894 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:30.930860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:30.930963 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:30.930939    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:31.150819 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:31.150784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:52:32.171723 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:32.171668    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="e6e2695c82c9a977d503b44c504f1d9b0a90c5efc283e280d50a62a20ab53a8a" exitCode=0
Sep 09 20:52:32.172418 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:32.172229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"e6e2695c82c9a977d503b44c504f1d9b0a90c5efc283e280d50a62a20ab53a8a"}
Sep 09 20:52:33.024297 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:33.024262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:33.024297 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:33.024294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:33.024565 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:33.024272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:33.024565 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:33.024383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:33.024565 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:33.024439    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:33.024565 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:33.024502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:34.126154 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.126107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:34.178455 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.178371    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="5d883dad7865bf7b119b2baf1b10606e3f68a35063040c451203f0a70fc7ad63" exitCode=0
Sep 09 20:52:34.178455 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.178413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"5d883dad7865bf7b119b2baf1b10606e3f68a35063040c451203f0a70fc7ad63"}
Sep 09 20:52:34.205819 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.205786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:34.663472 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.663440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:34.663775 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.663485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:34.663775 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.663587    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:34.663775 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.663647    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret podName:e35bdaed-546d-4756-ae8b-d4fa6ed1aab1 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.663627285 +0000 UTC m=+66.159901304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret") pod "global-pull-secret-syncer-qqzz4" (UID: "e35bdaed-546d-4756-ae8b-d4fa6ed1aab1") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:34.663775 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.663587    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:34.663775 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.663739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.663725814 +0000 UTC m=+66.159999834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:34.864060 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:34.864029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:34.864238 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.864221    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:34.864304 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.864248    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:34.864304 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.864261    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jfjxq for pod openshift-network-diagnostics/network-check-target-qn2gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:34.864404 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:34.864326    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq podName:7ec06c5e-3aef-4ea9-95a2-b4efe9f42365 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.864306106 +0000 UTC m=+66.360580149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfjxq" (UniqueName: "kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq") pod "network-check-target-qn2gw" (UID: "7ec06c5e-3aef-4ea9-95a2-b4efe9f42365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:35.025023 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.024945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:35.025177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.024945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:35.025177 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:35.025087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzt4v" podUID="6ab21338-9de8-4bef-a83e-5e42f15a438b"
Sep 09 20:52:35.025177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.024953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:35.025343 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:35.025122    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn2gw" podUID="7ec06c5e-3aef-4ea9-95a2-b4efe9f42365"
Sep 09 20:52:35.025343 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:35.025225    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qqzz4" podUID="e35bdaed-546d-4756-ae8b-d4fa6ed1aab1"
Sep 09 20:52:35.255875 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.255844    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-145.ec2.internal" event="NodeReady"
Sep 09 20:52:35.256248 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.256011    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:52:35.305569 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.305491    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n98lw"]
Sep 09 20:52:35.322427 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.322395    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rd5h"]
Sep 09 20:52:35.322639 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.322530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.325518 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.325494    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.325718 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.325679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.325842 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.325728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:52:35.326054 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.326038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gtzr8\""
Sep 09 20:52:35.348500 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.348468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n98lw"]
Sep 09 20:52:35.348500 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.348507    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rd5h"]
Sep 09 20:52:35.348655 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.348579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.351176 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.351159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:52:35.351373 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.351358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktpgk\""
Sep 09 20:52:35.351417 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.351358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:52:35.432401 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.432370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:35.468910 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.468871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tv2\" (UniqueName: \"kubernetes.io/projected/6b479bfe-ed66-450b-9f46-980dc2d8de9f-kube-api-access-q6tv2\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.469082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.468925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f7ee11-afe2-4227-a6e0-a77d8e392718-cert\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.469082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.468974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwtp\" (UniqueName: \"kubernetes.io/projected/50f7ee11-afe2-4227-a6e0-a77d8e392718-kube-api-access-hjwtp\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.469082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.469038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b479bfe-ed66-450b-9f46-980dc2d8de9f-config-volume\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.469082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.469056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.469293 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.469096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b479bfe-ed66-450b-9f46-980dc2d8de9f-tmp-dir\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.483895 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.483869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:35.570080 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.569991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b479bfe-ed66-450b-9f46-980dc2d8de9f-config-volume\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.570080 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.570302 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:35.570157    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:35.570302 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b479bfe-ed66-450b-9f46-980dc2d8de9f-tmp-dir\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.570302 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:35.570229    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.070210574 +0000 UTC m=+35.566484594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:35.570302 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tv2\" (UniqueName: \"kubernetes.io/projected/6b479bfe-ed66-450b-9f46-980dc2d8de9f-kube-api-access-q6tv2\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.570505 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f7ee11-afe2-4227-a6e0-a77d8e392718-cert\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.570505 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwtp\" (UniqueName: \"kubernetes.io/projected/50f7ee11-afe2-4227-a6e0-a77d8e392718-kube-api-access-hjwtp\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.570505 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b479bfe-ed66-450b-9f46-980dc2d8de9f-tmp-dir\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.570633 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.570619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b479bfe-ed66-450b-9f46-980dc2d8de9f-config-volume\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.575542 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.575522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f7ee11-afe2-4227-a6e0-a77d8e392718-cert\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.585813 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.585790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tv2\" (UniqueName: \"kubernetes.io/projected/6b479bfe-ed66-450b-9f46-980dc2d8de9f-kube-api-access-q6tv2\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:35.586784 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.586753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwtp\" (UniqueName: \"kubernetes.io/projected/50f7ee11-afe2-4227-a6e0-a77d8e392718-kube-api-access-hjwtp\") pod \"ingress-canary-n98lw\" (UID: \"50f7ee11-afe2-4227-a6e0-a77d8e392718\") " pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.635605 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.635573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n98lw"
Sep 09 20:52:35.960307 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:35.960106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n98lw"]
Sep 09 20:52:35.964619 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:35.964591    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f7ee11_afe2_4227_a6e0_a77d8e392718.slice/crio-fc0f64acc00e00e626916d0f8ad8d88ac85459fc738252f669da82224536064a WatchSource:0}: Error finding container fc0f64acc00e00e626916d0f8ad8d88ac85459fc738252f669da82224536064a: Status 404 returned error can't find the container with id fc0f64acc00e00e626916d0f8ad8d88ac85459fc738252f669da82224536064a
Sep 09 20:52:36.075347 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:36.075295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:36.075528 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:36.075470    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:36.075589 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:36.075563    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:37.075539176 +0000 UTC m=+36.571813219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:36.185347 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:36.185284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n98lw" event={"ID":"50f7ee11-afe2-4227-a6e0-a77d8e392718","Type":"ContainerStarted","Data":"fc0f64acc00e00e626916d0f8ad8d88ac85459fc738252f669da82224536064a"}
Sep 09 20:52:37.024727 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.024498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:52:37.024727 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.024498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:52:37.025344 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.024798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:52:37.027221 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.027117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74tsk\""
Sep 09 20:52:37.027221 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.027159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:52:37.028582 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.028184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:52:37.028582 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.028262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:52:37.028582 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.028427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:52:37.028830 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.028598    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2kpj\""
Sep 09 20:52:37.083377 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:37.083341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:37.083596 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:37.083580    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:37.083671 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:37.083654    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:39.083632873 +0000 UTC m=+38.579906916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:39.101250 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:39.101214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:39.101767 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:39.101392    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:39.101767 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:39.101472    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:43.101451195 +0000 UTC m=+42.597725222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:40.953551 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:40.953354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:41.015142 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:41.015101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:41.198592 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:41.198560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n98lw" event={"ID":"50f7ee11-afe2-4227-a6e0-a77d8e392718","Type":"ContainerStarted","Data":"ae9d0f3bb3ae68951dfa9a54f05ff9567ef9a83e3a62ab2541d1b24721865d0a"}
Sep 09 20:52:41.201258 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:41.201235    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="c2d4dca5011e653145eadde97a45e1386743975517a4036e2bc5ad4943a26962" exitCode=0
Sep 09 20:52:41.201388 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:41.201273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"c2d4dca5011e653145eadde97a45e1386743975517a4036e2bc5ad4943a26962"}
Sep 09 20:52:41.218414 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:41.218363    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n98lw" podStartSLOduration=2.000293569 podStartE2EDuration="6.218348717s" totalImagesPullingTime="4.218055148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:41.21671704 +0000 UTC m=+40.712991093" watchObservedRunningTime="2026-09-09 20:52:41.218348717 +0000 UTC m=+40.714622759"
Sep 09 20:52:42.206953 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:42.206912    2581 generic.go:356] "Generic (PLEG): container finished" podID="287066b7-5bb3-4553-9818-ffd2b8840bf9" containerID="c52b034ef03125f8b30d94d1384317366b5dbe1ad723a81cbc9b9d694c2aa0fd" exitCode=0
Sep 09 20:52:42.207379 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:42.206998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerDied","Data":"c52b034ef03125f8b30d94d1384317366b5dbe1ad723a81cbc9b9d694c2aa0fd"}
Sep 09 20:52:43.130438 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:43.130400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:43.130613 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:43.130535    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:43.130613 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:43.130587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.130573908 +0000 UTC m=+50.626847928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:43.212946 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:43.212905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vpgql" event={"ID":"287066b7-5bb3-4553-9818-ffd2b8840bf9","Type":"ContainerStarted","Data":"3e50b2f576afe56c1fe5351b905fdfbdc040050113ae200d6e7065a0ebff7fab"}
Sep 09 20:52:43.244619 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:43.244569    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vpgql" podStartSLOduration=8.371328959 podStartE2EDuration="41.244550916s" totalImagesPullingTime="32.873221957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:43.243854813 +0000 UTC m=+42.740128877" watchObservedRunningTime="2026-09-09 20:52:43.244550916 +0000 UTC m=+42.740824957"
Sep 09 20:52:44.628623 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.628589    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xnphq"]
Sep 09 20:52:44.665515 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.665484    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-87766bcc8-xnv4m"]
Sep 09 20:52:44.665678 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.665608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.668868 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.668845    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.669015 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.668996    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:52:44.669090 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.668997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w5j9g\""
Sep 09 20:52:44.670045 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.670026    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:52:44.670181 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.670164    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.675189 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.675171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:52:44.679386 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.679369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xnphq"]
Sep 09 20:52:44.679440 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.679397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-87766bcc8-xnv4m"]
Sep 09 20:52:44.679490 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.679474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.682070 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.682179 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:52:44.682376 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.682439 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lkg57\""
Sep 09 20:52:44.682492 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:52:44.682545 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:52:44.682595 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.682572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:52:44.733079 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.733049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"]
Sep 09 20:52:44.757531 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.757497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"]
Sep 09 20:52:44.757688 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.757602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:44.760251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.760219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.760251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.760219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.760488 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.760475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ftkdx\""
Sep 09 20:52:44.760540 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.760526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:52:44.843663 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.843840 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rts\" (UniqueName: \"kubernetes.io/projected/4425e76b-d0e7-4308-b2da-66eb66adcc39-kube-api-access-f4rts\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.843840 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-stats-auth\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.843840 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.843955 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.843955 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wg6\" (UniqueName: \"kubernetes.io/projected/44b96dff-bfb5-4403-8890-2225918ddc65-kube-api-access-57wg6\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.843955 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-tmp\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.843955 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-snapshots\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.844071 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.843998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-default-certificate\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.844071 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.844016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.844071 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.844067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4425e76b-d0e7-4308-b2da-66eb66adcc39-serving-cert\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945003 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.944960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-tmp\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945003 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-snapshots\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-default-certificate\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4425e76b-d0e7-4308-b2da-66eb66adcc39-serving-cert\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rts\" (UniqueName: \"kubernetes.io/projected/4425e76b-d0e7-4308-b2da-66eb66adcc39-kube-api-access-f4rts\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4v2\" (UniqueName: \"kubernetes.io/projected/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-kube-api-access-dc4v2\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:44.945233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-stats-auth\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.945593 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.945593 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945593 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wg6\" (UniqueName: \"kubernetes.io/projected/44b96dff-bfb5-4403-8890-2225918ddc65-kube-api-access-57wg6\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.945593 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:44.945572    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:44.945824 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:44.945640    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.44562025 +0000 UTC m=+44.941894284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:52:44.945824 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.945761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.945935 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:44.945896    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.445878563 +0000 UTC m=+44.942152610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:44.946098 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.946070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-tmp\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.946447 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.946421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4425e76b-d0e7-4308-b2da-66eb66adcc39-snapshots\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.946521 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.946451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4425e76b-d0e7-4308-b2da-66eb66adcc39-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.948140 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.948120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-default-certificate\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.948140 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.948129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4425e76b-d0e7-4308-b2da-66eb66adcc39-serving-cert\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.948259 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.948155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-stats-auth\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.956437 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.956404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wg6\" (UniqueName: \"kubernetes.io/projected/44b96dff-bfb5-4403-8890-2225918ddc65-kube-api-access-57wg6\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:44.956562 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.956480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rts\" (UniqueName: \"kubernetes.io/projected/4425e76b-d0e7-4308-b2da-66eb66adcc39-kube-api-access-f4rts\") pod \"insights-operator-76bcd6ccf5-xnphq\" (UID: \"4425e76b-d0e7-4308-b2da-66eb66adcc39\") " pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:44.975786 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:44.975749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq"
Sep 09 20:52:45.046425 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.046393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:45.046425 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.046430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4v2\" (UniqueName: \"kubernetes.io/projected/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-kube-api-access-dc4v2\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:45.046636 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.046602    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:45.046694 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.046676    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.546656354 +0000 UTC m=+45.042930387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:52:45.056368 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.056338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4v2\" (UniqueName: \"kubernetes.io/projected/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-kube-api-access-dc4v2\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:45.272994 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.272913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-xnphq"]
Sep 09 20:52:45.275646 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:45.275617    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4425e76b_d0e7_4308_b2da_66eb66adcc39.slice/crio-7b8a749116047402c580cd53e219970585d0f3e298318f9d9bc40d991dacfa3d WatchSource:0}: Error finding container 7b8a749116047402c580cd53e219970585d0f3e298318f9d9bc40d991dacfa3d: Status 404 returned error can't find the container with id 7b8a749116047402c580cd53e219970585d0f3e298318f9d9bc40d991dacfa3d
Sep 09 20:52:45.450445 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.450403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:45.450637 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.450487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:45.450637 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.450580    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:45.450637 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.450628    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.450608389 +0000 UTC m=+45.946882419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:45.450795 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.450657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.450638923 +0000 UTC m=+45.946912945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:52:45.550952 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.550862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:45.551062 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.551016    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:45.551100 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:45.551084    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.551063809 +0000 UTC m=+46.047337829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:52:45.614431 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.614393    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rbgh4"]
Sep 09 20:52:45.646252 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.646218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rbgh4"]
Sep 09 20:52:45.646660 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.646328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.649562 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.649537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:52:45.649562 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.649555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:52:45.649764 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.649540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:45.649973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.649955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x42np\""
Sep 09 20:52:45.650059 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.650036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:45.654638 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.654616    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:52:45.752197 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.752161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-config\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.752343 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.752218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/150c182f-77ba-4a14-9989-1fb90702c008-serving-cert\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.752343 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.752237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-trusted-ca\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.752439 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.752371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bx8m\" (UniqueName: \"kubernetes.io/projected/150c182f-77ba-4a14-9989-1fb90702c008-kube-api-access-8bx8m\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.852953 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.852852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bx8m\" (UniqueName: \"kubernetes.io/projected/150c182f-77ba-4a14-9989-1fb90702c008-kube-api-access-8bx8m\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.852953 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.852930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-config\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.853179 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.852992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/150c182f-77ba-4a14-9989-1fb90702c008-serving-cert\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.853179 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.853105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-trusted-ca\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.853717 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.853682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-config\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.854247 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.854230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/150c182f-77ba-4a14-9989-1fb90702c008-trusted-ca\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.855387 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.855363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/150c182f-77ba-4a14-9989-1fb90702c008-serving-cert\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.867411 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.867384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bx8m\" (UniqueName: \"kubernetes.io/projected/150c182f-77ba-4a14-9989-1fb90702c008-kube-api-access-8bx8m\") pod \"console-operator-85b454874b-rbgh4\" (UID: \"150c182f-77ba-4a14-9989-1fb90702c008\") " pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:45.955723 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:45.955680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:46.221995 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:46.221951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq" event={"ID":"4425e76b-d0e7-4308-b2da-66eb66adcc39","Type":"ContainerStarted","Data":"7b8a749116047402c580cd53e219970585d0f3e298318f9d9bc40d991dacfa3d"}
Sep 09 20:52:46.248420 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:46.248387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rbgh4"]
Sep 09 20:52:46.253962 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:46.253924    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c182f_77ba_4a14_9989_1fb90702c008.slice/crio-5c4588962ac64966179c5e759f50450973a9cdc74d58069e52436f656df5375a WatchSource:0}: Error finding container 5c4588962ac64966179c5e759f50450973a9cdc74d58069e52436f656df5375a: Status 404 returned error can't find the container with id 5c4588962ac64966179c5e759f50450973a9cdc74d58069e52436f656df5375a
Sep 09 20:52:46.457973 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:46.457935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:46.458144 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:46.458027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:46.458144 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:46.458103    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:46.458292 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:46.458169    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:48.458149488 +0000 UTC m=+47.954423522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:46.458292 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:46.458211    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:48.458180144 +0000 UTC m=+47.954454170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:52:46.558944 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:46.558854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:46.559085 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:46.559008    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:46.559085 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:46.559072    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:48.559056933 +0000 UTC m=+48.055330953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:52:47.226250 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:47.226203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" event={"ID":"150c182f-77ba-4a14-9989-1fb90702c008","Type":"ContainerStarted","Data":"5c4588962ac64966179c5e759f50450973a9cdc74d58069e52436f656df5375a"}
Sep 09 20:52:48.231313 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:48.231227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq" event={"ID":"4425e76b-d0e7-4308-b2da-66eb66adcc39","Type":"ContainerStarted","Data":"b66c5873daf9453b2bdd51e1629e592fcde6fe196bc2d46504b425ac0c11ea1d"}
Sep 09 20:52:48.250041 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:48.249989    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq" podStartSLOduration=1.5702270619999998 podStartE2EDuration="4.249969121s" totalImagesPullingTime="2.679742059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:48.249259605 +0000 UTC m=+47.745533649" watchObservedRunningTime="2026-09-09 20:52:48.249969121 +0000 UTC m=+47.746243166"
Sep 09 20:52:48.478276 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:48.478235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:48.478473 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:48.478328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:48.478473 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:48.478410    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:48.478579 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:48.478494    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.47847202 +0000 UTC m=+51.974746055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:52:48.478579 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:48.478518    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.478507622 +0000 UTC m=+51.974781642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:48.579625 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:48.579531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:48.579812 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:48.579733    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:48.579812 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:48.579803    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.579783379 +0000 UTC m=+52.076057406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:52:49.236352 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.236313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" event={"ID":"150c182f-77ba-4a14-9989-1fb90702c008","Type":"ContainerStarted","Data":"24a90ed0019b5e29ae86e1742a4902d736953137ca63b4a163ad434320d0e3a3"}
Sep 09 20:52:49.254867 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.254821    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" podStartSLOduration=1.426353855 podStartE2EDuration="4.254802991s" totalImagesPullingTime="2.828449136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:49.253910723 +0000 UTC m=+48.750184784" watchObservedRunningTime="2026-09-09 20:52:49.254802991 +0000 UTC m=+48.751077047"
Sep 09 20:52:49.658816 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.658778    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wlxqk"]
Sep 09 20:52:49.680008 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.679974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.683251 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.683221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wlxqk"]
Sep 09 20:52:49.689174 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.689149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4fce036-efab-41b6-9236-535aff9d4915-crio-socket\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.689286 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.689179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.689286 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.689195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.689286 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.689210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwfx\" (UniqueName: \"kubernetes.io/projected/c4fce036-efab-41b6-9236-535aff9d4915-kube-api-access-jhwfx\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.689410 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.689382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4fce036-efab-41b6-9236-535aff9d4915-data-volume\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.701024 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.701000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vkbt\""
Sep 09 20:52:49.701306 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.701292    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:52:49.701794 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.701777    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:52:49.790088 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4fce036-efab-41b6-9236-535aff9d4915-data-volume\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790243 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4fce036-efab-41b6-9236-535aff9d4915-crio-socket\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790243 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790243 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790243 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwfx\" (UniqueName: \"kubernetes.io/projected/c4fce036-efab-41b6-9236-535aff9d4915-kube-api-access-jhwfx\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790384 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:49.790246    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:49.790384 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:49.790311    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.290291462 +0000 UTC m=+49.786565495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:49.790384 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:49.790332    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:49.790384 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4fce036-efab-41b6-9236-535aff9d4915-crio-socket\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.790542 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:49.790412    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.290392752 +0000 UTC m=+49.786666777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:49.790542 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.790409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4fce036-efab-41b6-9236-535aff9d4915-data-volume\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:49.801680 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:49.801647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwfx\" (UniqueName: \"kubernetes.io/projected/c4fce036-efab-41b6-9236-535aff9d4915-kube-api-access-jhwfx\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:50.239984 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.239955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/0.log"
Sep 09 20:52:50.240385 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.239998    2581 generic.go:356] "Generic (PLEG): container finished" podID="150c182f-77ba-4a14-9989-1fb90702c008" containerID="24a90ed0019b5e29ae86e1742a4902d736953137ca63b4a163ad434320d0e3a3" exitCode=255
Sep 09 20:52:50.240385 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.240056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" event={"ID":"150c182f-77ba-4a14-9989-1fb90702c008","Type":"ContainerDied","Data":"24a90ed0019b5e29ae86e1742a4902d736953137ca63b4a163ad434320d0e3a3"}
Sep 09 20:52:50.240385 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.240370    2581 scope.go:117] "RemoveContainer" containerID="24a90ed0019b5e29ae86e1742a4902d736953137ca63b4a163ad434320d0e3a3"
Sep 09 20:52:50.293804 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.293776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:50.293917 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.293810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:50.293917 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:50.293887    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:50.294032 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:50.293948    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.293929952 +0000 UTC m=+50.790203973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:50.294093 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:50.294040    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:50.294144 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:50.294124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.294104376 +0000 UTC m=+50.790378416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:50.985503 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:50.985473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhfl6_7bb328a0-4ea0-4e63-9965-a10f79458da8/dns-node-resolver/0.log"
Sep 09 20:52:51.200631 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.200595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:52:51.200824 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.200760    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:51.200824 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.200819    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls podName:6b479bfe-ed66-450b-9f46-980dc2d8de9f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.200803268 +0000 UTC m=+66.697077293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls") pod "dns-default-7rd5h" (UID: "6b479bfe-ed66-450b-9f46-980dc2d8de9f") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:51.244322 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 20:52:51.244686 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/0.log"
Sep 09 20:52:51.244686 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244663    2581 generic.go:356] "Generic (PLEG): container finished" podID="150c182f-77ba-4a14-9989-1fb90702c008" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af" exitCode=255
Sep 09 20:52:51.244783 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" event={"ID":"150c182f-77ba-4a14-9989-1fb90702c008","Type":"ContainerDied","Data":"c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"}
Sep 09 20:52:51.244820 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244799    2581 scope.go:117] "RemoveContainer" containerID="24a90ed0019b5e29ae86e1742a4902d736953137ca63b4a163ad434320d0e3a3"
Sep 09 20:52:51.244995 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.244975    2581 scope.go:117] "RemoveContainer" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"
Sep 09 20:52:51.245194 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.245172    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rbgh4_openshift-console-operator(150c182f-77ba-4a14-9989-1fb90702c008)\"" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" podUID="150c182f-77ba-4a14-9989-1fb90702c008"
Sep 09 20:52:51.301801 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.301754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:51.301801 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.301807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:51.302015 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.301867    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:51.302015 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.301928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:53.301911535 +0000 UTC m=+52.798185561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:51.302015 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.301995    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:51.302141 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:51.302053    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:53.302035948 +0000 UTC m=+52.798309968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:51.598811 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.598731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"]
Sep 09 20:52:51.603631 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.603615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.606459 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.606433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m5x68\""
Sep 09 20:52:51.606459 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.606450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:52:51.606659 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.606472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:52:51.606659 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.606487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:51.606659 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.606443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:51.612536 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.612515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"]
Sep 09 20:52:51.704238 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.704197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gb6\" (UniqueName: \"kubernetes.io/projected/905f40ee-43b9-48ee-ab53-e6503d6aac84-kube-api-access-w6gb6\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.704428 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.704351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/905f40ee-43b9-48ee-ab53-e6503d6aac84-serving-cert\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.704494 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.704470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/905f40ee-43b9-48ee-ab53-e6503d6aac84-config\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.805694 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.805650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/905f40ee-43b9-48ee-ab53-e6503d6aac84-serving-cert\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.805913 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.805779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/905f40ee-43b9-48ee-ab53-e6503d6aac84-config\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.805913 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.805808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gb6\" (UniqueName: \"kubernetes.io/projected/905f40ee-43b9-48ee-ab53-e6503d6aac84-kube-api-access-w6gb6\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.806329 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.806308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/905f40ee-43b9-48ee-ab53-e6503d6aac84-config\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.807867 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.807844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/905f40ee-43b9-48ee-ab53-e6503d6aac84-serving-cert\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.813943 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.813925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gb6\" (UniqueName: \"kubernetes.io/projected/905f40ee-43b9-48ee-ab53-e6503d6aac84-kube-api-access-w6gb6\") pod \"service-ca-operator-859f94c4b-gl642\" (UID: \"905f40ee-43b9-48ee-ab53-e6503d6aac84\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:51.912135 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:51.912103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"
Sep 09 20:52:52.178883 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.178805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642"]
Sep 09 20:52:52.181875 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:52:52.181854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905f40ee_43b9_48ee_ab53_e6503d6aac84.slice/crio-30c4e10d1080d80e695530dbd90bfbdbbede4e1bb4d21a84859ebc11edb5b8be WatchSource:0}: Error finding container 30c4e10d1080d80e695530dbd90bfbdbbede4e1bb4d21a84859ebc11edb5b8be: Status 404 returned error can't find the container with id 30c4e10d1080d80e695530dbd90bfbdbbede4e1bb4d21a84859ebc11edb5b8be
Sep 09 20:52:52.184328 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.184287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ppvcw_75d33de8-9ddb-4f97-9691-1ee5220ba6cb/node-ca/0.log"
Sep 09 20:52:52.252016 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.251988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 20:52:52.252408 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.252350    2581 scope.go:117] "RemoveContainer" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"
Sep 09 20:52:52.252560 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.252533    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rbgh4_openshift-console-operator(150c182f-77ba-4a14-9989-1fb90702c008)\"" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" podUID="150c182f-77ba-4a14-9989-1fb90702c008"
Sep 09 20:52:52.253377 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.253352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642" event={"ID":"905f40ee-43b9-48ee-ab53-e6503d6aac84","Type":"ContainerStarted","Data":"30c4e10d1080d80e695530dbd90bfbdbbede4e1bb4d21a84859ebc11edb5b8be"}
Sep 09 20:52:52.510796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.510688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:52.510796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.510764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:52:52.510977 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.510829    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:52.510977 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.510898    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:00.510882223 +0000 UTC m=+60.007156244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:52:52.510977 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.510913    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:00.51090735 +0000 UTC m=+60.007181370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:52.611362 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.611321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:52:52.611520 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.611496    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:52.611570 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:52.611560    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:00.611543531 +0000 UTC m=+60.107817551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:52:52.784394 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:52.784318    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n98lw_50f7ee11-afe2-4227-a6e0-a77d8e392718/serve-healthcheck-canary/0.log"
Sep 09 20:52:53.318890 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:53.318852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:53.318890 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:53.318892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:53.319410 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:53.319031    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:53.319410 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:53.319101    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.319079409 +0000 UTC m=+56.815353431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:53.319410 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:53.319130    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:53.319410 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:53.319188    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.319169265 +0000 UTC m=+56.815443291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:55.263324 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:55.263289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642" event={"ID":"905f40ee-43b9-48ee-ab53-e6503d6aac84","Type":"ContainerStarted","Data":"be5b172fd467d479b4def28b96793ad642ea25bca5c2ae26e66878d5f8399d9f"}
Sep 09 20:52:55.956421 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:55.956387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:55.956789 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:55.956775    2581 scope.go:117] "RemoveContainer" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"
Sep 09 20:52:55.956933 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:55.956915    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rbgh4_openshift-console-operator(150c182f-77ba-4a14-9989-1fb90702c008)\"" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" podUID="150c182f-77ba-4a14-9989-1fb90702c008"
Sep 09 20:52:57.358098 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:57.358059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:57.358539 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:57.358113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:52:57.358539 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:57.358218    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:57.358539 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:57.358284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.358267817 +0000 UTC m=+64.854541838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:57.358539 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:57.358290    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:57.358539 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:57.358358    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.35833889 +0000 UTC m=+64.854612913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:59.237095 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:59.237054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:52:59.237497 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:52:59.237467    2581 scope.go:117] "RemoveContainer" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"
Sep 09 20:52:59.237631 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:52:59.237611    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rbgh4_openshift-console-operator(150c182f-77ba-4a14-9989-1fb90702c008)\"" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" podUID="150c182f-77ba-4a14-9989-1fb90702c008"
Sep 09 20:53:00.583593 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:00.583540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:00.584042 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:00.583618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:00.584042 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:00.583694    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:53:00.584042 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:00.583785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.58376889 +0000 UTC m=+76.080042910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : secret "router-metrics-certs-default" not found
Sep 09 20:53:00.584042 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:00.583800    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle podName:44b96dff-bfb5-4403-8890-2225918ddc65 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.583793451 +0000 UTC m=+76.080067471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle") pod "router-default-87766bcc8-xnv4m" (UID: "44b96dff-bfb5-4403-8890-2225918ddc65") : configmap references non-existent config key: service-ca.crt
Sep 09 20:53:00.684677 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:00.684639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:53:00.684858 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:00.684802    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:53:00.684900 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:00.684872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls podName:ab8dd806-0d2a-49a2-a3f7-2eea504c73f8 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.684852529 +0000 UTC m=+76.181126566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bgr4m" (UID: "ab8dd806-0d2a-49a2-a3f7-2eea504c73f8") : secret "samples-operator-tls" not found
Sep 09 20:53:02.188924 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:02.188894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zp8rd"
Sep 09 20:53:02.218056 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:02.217996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642" podStartSLOduration=8.669272276000001 podStartE2EDuration="11.217977684s" totalImagesPullingTime="2.548705408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.280462325 +0000 UTC m=+54.776736367" watchObservedRunningTime="2026-09-09 20:53:02.217977684 +0000 UTC m=+61.714251740"
Sep 09 20:53:05.422175 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:05.422136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:05.422549 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:05.422190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:05.422549 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:05.422288    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.422549 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:05.422389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls podName:c4fce036-efab-41b6-9236-535aff9d4915 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.422370514 +0000 UTC m=+80.918644538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wlxqk" (UID: "c4fce036-efab-41b6-9236-535aff9d4915") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.422678 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:05.422662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4fce036-efab-41b6-9236-535aff9d4915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:06.732855 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.732807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:53:06.733363 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.732927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:53:06.736601 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.736578    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:06.736673 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.736610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:06.743866 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:06.743842    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:53:06.743934 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:06.743909    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs podName:6ab21338-9de8-4bef-a83e-5e42f15a438b nodeName:}" failed. No retries permitted until 2026-09-09 20:54:10.743891436 +0000 UTC m=+130.240165455 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs") pod "network-metrics-daemon-bzt4v" (UID: "6ab21338-9de8-4bef-a83e-5e42f15a438b") : secret "metrics-daemon-secret" not found
Sep 09 20:53:06.746005 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.745981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e35bdaed-546d-4756-ae8b-d4fa6ed1aab1-original-pull-secret\") pod \"global-pull-secret-syncer-qqzz4\" (UID: \"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1\") " pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:53:06.746904 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.746879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qqzz4"
Sep 09 20:53:06.935384 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.935326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:53:06.938253 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.938232    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:06.948173 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.948148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:06.959071 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:06.959048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjxq\" (UniqueName: \"kubernetes.io/projected/7ec06c5e-3aef-4ea9-95a2-b4efe9f42365-kube-api-access-jfjxq\") pod \"network-check-target-qn2gw\" (UID: \"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365\") " pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:53:07.017233 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.017156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qqzz4"]
Sep 09 20:53:07.021642 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:07.021609    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35bdaed_546d_4756_ae8b_d4fa6ed1aab1.slice/crio-81cd067c0f3d16a6ee3116ef18cc472ecd6fb191fd8a5e82b8fcbba48b617334 WatchSource:0}: Error finding container 81cd067c0f3d16a6ee3116ef18cc472ecd6fb191fd8a5e82b8fcbba48b617334: Status 404 returned error can't find the container with id 81cd067c0f3d16a6ee3116ef18cc472ecd6fb191fd8a5e82b8fcbba48b617334
Sep 09 20:53:07.041972 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.041946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74tsk\""
Sep 09 20:53:07.050291 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.050269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:53:07.238366 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.238326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:53:07.240605 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.240580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b479bfe-ed66-450b-9f46-980dc2d8de9f-metrics-tls\") pod \"dns-default-7rd5h\" (UID: \"6b479bfe-ed66-450b-9f46-980dc2d8de9f\") " pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:53:07.293032 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.292947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qqzz4" event={"ID":"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1","Type":"ContainerStarted","Data":"81cd067c0f3d16a6ee3116ef18cc472ecd6fb191fd8a5e82b8fcbba48b617334"}
Sep 09 20:53:07.327261 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.327231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qn2gw"]
Sep 09 20:53:07.330479 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:07.330448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec06c5e_3aef_4ea9_95a2_b4efe9f42365.slice/crio-846e712c89fb483ff75b22664e6c903e42c1bf1a0aed8801ce077cd2730a2122 WatchSource:0}: Error finding container 846e712c89fb483ff75b22664e6c903e42c1bf1a0aed8801ce077cd2730a2122: Status 404 returned error can't find the container with id 846e712c89fb483ff75b22664e6c903e42c1bf1a0aed8801ce077cd2730a2122
Sep 09 20:53:07.460736 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.460687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktpgk\""
Sep 09 20:53:07.467662 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.467635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:53:07.753764 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:07.753712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rd5h"]
Sep 09 20:53:07.758660 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:07.758605    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b479bfe_ed66_450b_9f46_980dc2d8de9f.slice/crio-ae3d5b1377a4dd6206a0f7d38ab2dc4cd65968d99819bbf4536370c551b85b3c WatchSource:0}: Error finding container ae3d5b1377a4dd6206a0f7d38ab2dc4cd65968d99819bbf4536370c551b85b3c: Status 404 returned error can't find the container with id ae3d5b1377a4dd6206a0f7d38ab2dc4cd65968d99819bbf4536370c551b85b3c
Sep 09 20:53:08.298952 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:08.298874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rd5h" event={"ID":"6b479bfe-ed66-450b-9f46-980dc2d8de9f","Type":"ContainerStarted","Data":"ae3d5b1377a4dd6206a0f7d38ab2dc4cd65968d99819bbf4536370c551b85b3c"}
Sep 09 20:53:08.300349 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:08.300321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qn2gw" event={"ID":"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365","Type":"ContainerStarted","Data":"846e712c89fb483ff75b22664e6c903e42c1bf1a0aed8801ce077cd2730a2122"}
Sep 09 20:53:12.024915 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:12.024868    2581 scope.go:117] "RemoveContainer" containerID="c4d693254da690f97447c8c54d864dc235169341655a44b954e12e161e5091af"
Sep 09 20:53:13.318015 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.317986    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 20:53:13.318481 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.318083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rbgh4" event={"ID":"150c182f-77ba-4a14-9989-1fb90702c008","Type":"ContainerStarted","Data":"1f155b420d422aa16b09ce20f06b99939f88ff971c7dadd9ce74974b91c6c612"}
Sep 09 20:53:13.318481 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.318384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:53:13.319992 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.319962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rd5h" event={"ID":"6b479bfe-ed66-450b-9f46-980dc2d8de9f","Type":"ContainerStarted","Data":"82d7cec7f6c89f36dd2f45c471c4b26443f79ff48a001ab6d07aab144bc57942"}
Sep 09 20:53:13.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.320006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rd5h" event={"ID":"6b479bfe-ed66-450b-9f46-980dc2d8de9f","Type":"ContainerStarted","Data":"47d19ac720fde8580d7436d197e5c2e839c514751af0ff6833832f4a9e94cd88"}
Sep 09 20:53:13.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.320049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:53:13.321634 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.321599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qn2gw" event={"ID":"7ec06c5e-3aef-4ea9-95a2-b4efe9f42365","Type":"ContainerStarted","Data":"2ec2429a7125a669399205b2da2223e4b08b06b646d4e169e4e007fc21914a2c"}
Sep 09 20:53:13.321770 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.321748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:53:13.323165 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.323134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qqzz4" event={"ID":"e35bdaed-546d-4756-ae8b-d4fa6ed1aab1","Type":"ContainerStarted","Data":"c58d193b31edfa4591ed08b56b699fe0675c1e2051698342c3cbc5b30bf763a4"}
Sep 09 20:53:13.323795 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.323776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-rbgh4"
Sep 09 20:53:13.359329 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.359253    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7rd5h" podStartSLOduration=33.688892884 podStartE2EDuration="38.359232032s" totalImagesPullingTime="4.670339148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:13.357983211 +0000 UTC m=+72.854257253" watchObservedRunningTime="2026-09-09 20:53:13.359232032 +0000 UTC m=+72.855506074"
Sep 09 20:53:13.396553 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.396495    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qqzz4" podStartSLOduration=65.983595497 podStartE2EDuration="1m11.396479368s" totalImagesPullingTime="5.412883871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:13.378135091 +0000 UTC m=+72.874409143" watchObservedRunningTime="2026-09-09 20:53:13.396479368 +0000 UTC m=+72.892753410"
Sep 09 20:53:13.416253 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:13.416193    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qn2gw" podStartSLOduration=66.321131783 podStartE2EDuration="1m11.416173125s" totalImagesPullingTime="5.095041342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:13.41535675 +0000 UTC m=+72.911630811" watchObservedRunningTime="2026-09-09 20:53:13.416173125 +0000 UTC m=+72.912447169"
Sep 09 20:53:16.614006 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.613962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:16.614006 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.614013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:16.614562 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.614543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b96dff-bfb5-4403-8890-2225918ddc65-service-ca-bundle\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:16.616259 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.616234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b96dff-bfb5-4403-8890-2225918ddc65-metrics-certs\") pod \"router-default-87766bcc8-xnv4m\" (UID: \"44b96dff-bfb5-4403-8890-2225918ddc65\") " pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:16.715356 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.715321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:53:16.717813 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.717779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab8dd806-0d2a-49a2-a3f7-2eea504c73f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bgr4m\" (UID: \"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:53:16.791109 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.791080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lkg57\""
Sep 09 20:53:16.799367 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.799348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:16.868997 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.868923    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ftkdx\""
Sep 09 20:53:16.877075 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:16.877040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"
Sep 09 20:53:17.076032 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.075976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-87766bcc8-xnv4m"]
Sep 09 20:53:17.077916 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:17.077892    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b96dff_bfb5_4403_8890_2225918ddc65.slice/crio-9cab98e93bb67d2f24f2f1ec3134284d17b0bca73a2d7302a67164da0eaf133d WatchSource:0}: Error finding container 9cab98e93bb67d2f24f2f1ec3134284d17b0bca73a2d7302a67164da0eaf133d: Status 404 returned error can't find the container with id 9cab98e93bb67d2f24f2f1ec3134284d17b0bca73a2d7302a67164da0eaf133d
Sep 09 20:53:17.150869 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.150843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m"]
Sep 09 20:53:17.339228 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.339194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m" event={"ID":"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8","Type":"ContainerStarted","Data":"2c18969a5c585e2029a757befddf6e528a5a8e2a673ccb5ecca0ab5fc286b2cc"}
Sep 09 20:53:17.340580 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.340557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-87766bcc8-xnv4m" event={"ID":"44b96dff-bfb5-4403-8890-2225918ddc65","Type":"ContainerStarted","Data":"de17beaa33520adec6b7197b2f062691f38739bec92da097076432f0f73c5f3e"}
Sep 09 20:53:17.340694 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.340593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-87766bcc8-xnv4m" event={"ID":"44b96dff-bfb5-4403-8890-2225918ddc65","Type":"ContainerStarted","Data":"9cab98e93bb67d2f24f2f1ec3134284d17b0bca73a2d7302a67164da0eaf133d"}
Sep 09 20:53:17.364339 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.364290    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-87766bcc8-xnv4m" podStartSLOduration=33.364274133 podStartE2EDuration="33.364274133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:17.36305432 +0000 UTC m=+76.859328374" watchObservedRunningTime="2026-09-09 20:53:17.364274133 +0000 UTC m=+76.860548174"
Sep 09 20:53:17.799599 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.799565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:17.802193 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:17.802162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:18.344328 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.344292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:18.345485 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.345464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-87766bcc8-xnv4m"
Sep 09 20:53:18.614001 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.613920    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"]
Sep 09 20:53:18.618018 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.617993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.622488 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.622459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:53:18.623016 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.622997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:53:18.623226 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.623052    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:53:18.623226 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.623078    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:53:18.623226 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.623123    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:53:18.623552 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.623381    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:53:18.623657 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.623633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:53:18.638628 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.638600    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"]
Sep 09 20:53:18.734990 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.734951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.735177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.734999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.735177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.735032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.735177 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.735079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.735319 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.735182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrgt\" (UniqueName: \"kubernetes.io/projected/e49384fd-0220-4ca1-990c-863fdf9f164e-kube-api-access-fzrgt\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.735319 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.735246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49384fd-0220-4ca1-990c-863fdf9f164e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.836507 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrgt\" (UniqueName: \"kubernetes.io/projected/e49384fd-0220-4ca1-990c-863fdf9f164e-kube-api-access-fzrgt\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837007 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49384fd-0220-4ca1-990c-863fdf9f164e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837007 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837007 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837007 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837007 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.836619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.837450 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.837392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49384fd-0220-4ca1-990c-863fdf9f164e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.839444 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.839419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.839674 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.839626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-ca\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.839674 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.839644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.839819 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.839722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e49384fd-0220-4ca1-990c-863fdf9f164e-hub\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.846716 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.846679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrgt\" (UniqueName: \"kubernetes.io/projected/e49384fd-0220-4ca1-990c-863fdf9f164e-kube-api-access-fzrgt\") pod \"cluster-proxy-proxy-agent-f77b5c87-jr2cq\" (UID: \"e49384fd-0220-4ca1-990c-863fdf9f164e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:18.941380 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:18.941341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"
Sep 09 20:53:19.217914 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:19.217682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq"]
Sep 09 20:53:19.523467 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:19.523394    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49384fd_0220_4ca1_990c_863fdf9f164e.slice/crio-8bbdde28032406607c1c55a06f1638288af9fb0eb1b763bd8703805b3f01f714 WatchSource:0}: Error finding container 8bbdde28032406607c1c55a06f1638288af9fb0eb1b763bd8703805b3f01f714: Status 404 returned error can't find the container with id 8bbdde28032406607c1c55a06f1638288af9fb0eb1b763bd8703805b3f01f714
Sep 09 20:53:20.353243 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:20.353209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m" event={"ID":"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8","Type":"ContainerStarted","Data":"e06d9b5887e1085712e59e13691c68ee0c2b6e24599a70341612f50176a01371"}
Sep 09 20:53:20.353762 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:20.353268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m" event={"ID":"ab8dd806-0d2a-49a2-a3f7-2eea504c73f8","Type":"ContainerStarted","Data":"ae1919f1b0ebe5f6aa14d8a6dd8d00af2280d4fec6f6bf65424f4ed87cbbbd12"}
Sep 09 20:53:20.354526 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:20.354504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq" event={"ID":"e49384fd-0220-4ca1-990c-863fdf9f164e","Type":"ContainerStarted","Data":"8bbdde28032406607c1c55a06f1638288af9fb0eb1b763bd8703805b3f01f714"}
Sep 09 20:53:20.371793 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:20.371740    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bgr4m" podStartSLOduration=33.980232897 podStartE2EDuration="36.371721139s" totalImagesPullingTime="2.391488242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:20.370805977 +0000 UTC m=+79.867080045" watchObservedRunningTime="2026-09-09 20:53:20.371721139 +0000 UTC m=+79.867995172"
Sep 09 20:53:21.455686 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:21.455627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:21.458914 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:21.458884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4fce036-efab-41b6-9236-535aff9d4915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wlxqk\" (UID: \"c4fce036-efab-41b6-9236-535aff9d4915\") " pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:21.492531 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:21.492485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vkbt\""
Sep 09 20:53:21.500134 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:21.500071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wlxqk"
Sep 09 20:53:21.790381 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:21.790298    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wlxqk"]
Sep 09 20:53:22.519551 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:22.519517    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fce036_efab_41b6_9236_535aff9d4915.slice/crio-67471a51fdc47c72c66860ef4fff9d37c6293194e2d056202433268336a586fe WatchSource:0}: Error finding container 67471a51fdc47c72c66860ef4fff9d37c6293194e2d056202433268336a586fe: Status 404 returned error can't find the container with id 67471a51fdc47c72c66860ef4fff9d37c6293194e2d056202433268336a586fe
Sep 09 20:53:23.330694 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:23.330573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rd5h"
Sep 09 20:53:23.368632 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:23.368594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlxqk" event={"ID":"c4fce036-efab-41b6-9236-535aff9d4915","Type":"ContainerStarted","Data":"85a07beb4c6f4cc4e290c569658637ebc2d95aa5541d4351b2a8bd952c7951d3"}
Sep 09 20:53:23.368824 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:23.368668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlxqk" event={"ID":"c4fce036-efab-41b6-9236-535aff9d4915","Type":"ContainerStarted","Data":"67471a51fdc47c72c66860ef4fff9d37c6293194e2d056202433268336a586fe"}
Sep 09 20:53:24.374597 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:24.374557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlxqk" event={"ID":"c4fce036-efab-41b6-9236-535aff9d4915","Type":"ContainerStarted","Data":"6e542a593405edc695b8eed8f4e6a3975d5a7edea2f978f510b279a13685e19b"}
Sep 09 20:53:24.376768 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:24.376742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq" event={"ID":"e49384fd-0220-4ca1-990c-863fdf9f164e","Type":"ContainerStarted","Data":"cef5636f4146575748035171aa06973b56d926d4eeae9ad9fc09df5994a2d2fe"}
Sep 09 20:53:24.376896 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:24.376783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq" event={"ID":"e49384fd-0220-4ca1-990c-863fdf9f164e","Type":"ContainerStarted","Data":"0fe0e2a746c9ec6f944ced3791acd3c0d3f4993aab3408afd29f930086ca6a5e"}
Sep 09 20:53:24.376896 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:24.376801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq" event={"ID":"e49384fd-0220-4ca1-990c-863fdf9f164e","Type":"ContainerStarted","Data":"ff61d43aaf1f60be658dee095650a6d0f42a54fdc2d7b0aa00031f80dd8074e2"}
Sep 09 20:53:24.402240 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:24.402191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f77b5c87-jr2cq" podStartSLOduration=2.04343545 podStartE2EDuration="6.402173331s" totalImagesPullingTime="4.358737881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.401459427 +0000 UTC m=+83.897733468" watchObservedRunningTime="2026-09-09 20:53:24.402173331 +0000 UTC m=+83.898447420"
Sep 09 20:53:26.385986 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:26.385942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wlxqk" event={"ID":"c4fce036-efab-41b6-9236-535aff9d4915","Type":"ContainerStarted","Data":"c2d4348508b384e0bb8dc166157af18dfef7ba7aea2a74e52b163fee7860ce31"}
Sep 09 20:53:26.406786 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:26.406733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wlxqk" podStartSLOduration=34.704164775 podStartE2EDuration="37.406695339s" totalImagesPullingTime="2.702530564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:26.405513038 +0000 UTC m=+85.901787098" watchObservedRunningTime="2026-09-09 20:53:26.406695339 +0000 UTC m=+85.902969382"
Sep 09 20:53:31.214133 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.214103    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sv6df"]
Sep 09 20:53:31.219568 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.219549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.223454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.223428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:31.223575 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.223434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:31.223575 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.223477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:31.224499 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.224481    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:31.224566 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.224500    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjj4f\""
Sep 09 20:53:31.224566 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.224536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:53:31.224652 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.224574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:53:31.333570 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333771 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-root\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333771 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-textfile\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333771 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-sys\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333771 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-metrics-client-ca\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333907 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-wtmp\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333907 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8d7\" (UniqueName: \"kubernetes.io/projected/a649a47c-7865-48b1-8cf8-0d7a06e6185e-kube-api-access-7q8d7\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333907 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.333907 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.333882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434265 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-root\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-textfile\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-sys\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-metrics-client-ca\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-root\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434454 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-wtmp\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:31.434466    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8d7\" (UniqueName: \"kubernetes.io/projected/a649a47c-7865-48b1-8cf8-0d7a06e6185e-kube-api-access-7q8d7\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:53:31.434540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls podName:a649a47c-7865-48b1-8cf8-0d7a06e6185e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:31.934518939 +0000 UTC m=+91.430792958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls") pod "node-exporter-sv6df" (UID: "a649a47c-7865-48b1-8cf8-0d7a06e6185e") : secret "node-exporter-tls" not found
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-wtmp\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649a47c-7865-48b1-8cf8-0d7a06e6185e-sys\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.434880 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-textfile\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.435074 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.435074 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.434995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649a47c-7865-48b1-8cf8-0d7a06e6185e-metrics-client-ca\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.436642 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.436622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.443201 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.443173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8d7\" (UniqueName: \"kubernetes.io/projected/a649a47c-7865-48b1-8cf8-0d7a06e6185e-kube-api-access-7q8d7\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.938496 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.938453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:31.940823 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:31.940801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649a47c-7865-48b1-8cf8-0d7a06e6185e-node-exporter-tls\") pod \"node-exporter-sv6df\" (UID: \"a649a47c-7865-48b1-8cf8-0d7a06e6185e\") " pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:32.129357 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:32.129317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sv6df"
Sep 09 20:53:32.142416 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:32.142381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda649a47c_7865_48b1_8cf8_0d7a06e6185e.slice/crio-0a2fea2c83b7e65f236e9e57772f0717827d1afa835bed52cb37411c0e190d24 WatchSource:0}: Error finding container 0a2fea2c83b7e65f236e9e57772f0717827d1afa835bed52cb37411c0e190d24: Status 404 returned error can't find the container with id 0a2fea2c83b7e65f236e9e57772f0717827d1afa835bed52cb37411c0e190d24
Sep 09 20:53:32.408957 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:32.408907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sv6df" event={"ID":"a649a47c-7865-48b1-8cf8-0d7a06e6185e","Type":"ContainerStarted","Data":"0a2fea2c83b7e65f236e9e57772f0717827d1afa835bed52cb37411c0e190d24"}
Sep 09 20:53:34.417559 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:34.417526    2581 generic.go:356] "Generic (PLEG): container finished" podID="a649a47c-7865-48b1-8cf8-0d7a06e6185e" containerID="75b5a10959f06986c52329341de1f789ae165754a15aed618a014c7c47294e7b" exitCode=0
Sep 09 20:53:34.418082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:34.417595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sv6df" event={"ID":"a649a47c-7865-48b1-8cf8-0d7a06e6185e","Type":"ContainerDied","Data":"75b5a10959f06986c52329341de1f789ae165754a15aed618a014c7c47294e7b"}
Sep 09 20:53:35.422578 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:35.422540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sv6df" event={"ID":"a649a47c-7865-48b1-8cf8-0d7a06e6185e","Type":"ContainerStarted","Data":"d24ea6c60046a3bf465824a10e6619a3b9b741a9542acdcc6bc6fabdc61f64e9"}
Sep 09 20:53:35.423044 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:35.422593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sv6df" event={"ID":"a649a47c-7865-48b1-8cf8-0d7a06e6185e","Type":"ContainerStarted","Data":"79891220d8a93c25cb621017c908a313f3cb64e403f34d8b299c216bb0a07504"}
Sep 09 20:53:35.448861 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:35.448803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sv6df" podStartSLOduration=3.067428523 podStartE2EDuration="4.448785369s" totalImagesPullingTime="1.317981656s" totalInitContainerRuntime="63.37519ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:35.446884327 +0000 UTC m=+94.943158379" watchObservedRunningTime="2026-09-09 20:53:35.448785369 +0000 UTC m=+94.945059411"
Sep 09 20:53:40.791320 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.791239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:53:40.795410 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.795388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.799745 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.799715    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:40.799869 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.799754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l62dd\""
Sep 09 20:53:40.800957 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.800933    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:40.801061 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.801015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:40.801121 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.801088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:40.801121 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.801102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:40.801234 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.801218    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:40.802217 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.802200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:40.807442 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.807425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:40.812228 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.812207    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:53:40.814931 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.814910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815032 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.814939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbsv\" (UniqueName: \"kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815032 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.814960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815124 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.815039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815124 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.815075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815124 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.815092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.815124 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.815107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916005 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.915970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbsv\" (UniqueName: \"kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.916885 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.917014 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.917014 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.916934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.917182 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.917161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.918579 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.918557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.918651 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.918592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:40.925279 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:40.925252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbsv\" (UniqueName: \"kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv\") pod \"console-7b89757cf4-vpdkh\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") " pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:41.104853 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:41.104754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:41.376689 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:41.376660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:53:41.378856 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:53:41.378829    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffe4716_87f8_4306_91e5_b11d2828a72a.slice/crio-f37968335f2050930049756cb99e8c50c6c26d6be36143ca1347b114317355fd WatchSource:0}: Error finding container f37968335f2050930049756cb99e8c50c6c26d6be36143ca1347b114317355fd: Status 404 returned error can't find the container with id f37968335f2050930049756cb99e8c50c6c26d6be36143ca1347b114317355fd
Sep 09 20:53:41.443028 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:41.442991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b89757cf4-vpdkh" event={"ID":"affe4716-87f8-4306-91e5-b11d2828a72a","Type":"ContainerStarted","Data":"f37968335f2050930049756cb99e8c50c6c26d6be36143ca1347b114317355fd"}
Sep 09 20:53:44.331064 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:44.330982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qn2gw"
Sep 09 20:53:44.456825 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:44.456790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b89757cf4-vpdkh" event={"ID":"affe4716-87f8-4306-91e5-b11d2828a72a","Type":"ContainerStarted","Data":"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"}
Sep 09 20:53:44.545715 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:44.545655    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b89757cf4-vpdkh" podStartSLOduration=1.9026171330000001 podStartE2EDuration="4.545634335s" totalImagesPullingTime="2.643017202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:44.543808001 +0000 UTC m=+104.040082058" watchObservedRunningTime="2026-09-09 20:53:44.545634335 +0000 UTC m=+104.041908379"
Sep 09 20:53:51.105746 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:51.105692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:51.106332 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:51.105786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:51.110506 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:51.110481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:51.485208 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:51.485177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:53:59.511949 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:59.511914    2581 generic.go:356] "Generic (PLEG): container finished" podID="4425e76b-d0e7-4308-b2da-66eb66adcc39" containerID="b66c5873daf9453b2bdd51e1629e592fcde6fe196bc2d46504b425ac0c11ea1d" exitCode=0
Sep 09 20:53:59.512328 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:59.511990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq" event={"ID":"4425e76b-d0e7-4308-b2da-66eb66adcc39","Type":"ContainerDied","Data":"b66c5873daf9453b2bdd51e1629e592fcde6fe196bc2d46504b425ac0c11ea1d"}
Sep 09 20:53:59.512371 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:53:59.512346    2581 scope.go:117] "RemoveContainer" containerID="b66c5873daf9453b2bdd51e1629e592fcde6fe196bc2d46504b425ac0c11ea1d"
Sep 09 20:54:00.519485 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:00.519445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-xnphq" event={"ID":"4425e76b-d0e7-4308-b2da-66eb66adcc39","Type":"ContainerStarted","Data":"b8f82bc3e5e8ae094f254807986d434d8e676594c41055f58e739aa554a3afe0"}
Sep 09 20:54:10.555929 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.555886    2581 generic.go:356] "Generic (PLEG): container finished" podID="905f40ee-43b9-48ee-ab53-e6503d6aac84" containerID="be5b172fd467d479b4def28b96793ad642ea25bca5c2ae26e66878d5f8399d9f" exitCode=0
Sep 09 20:54:10.556367 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.555959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642" event={"ID":"905f40ee-43b9-48ee-ab53-e6503d6aac84","Type":"ContainerDied","Data":"be5b172fd467d479b4def28b96793ad642ea25bca5c2ae26e66878d5f8399d9f"}
Sep 09 20:54:10.556367 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.556266    2581 scope.go:117] "RemoveContainer" containerID="be5b172fd467d479b4def28b96793ad642ea25bca5c2ae26e66878d5f8399d9f"
Sep 09 20:54:10.757584 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.757543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:54:10.759936 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.759907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ab21338-9de8-4bef-a83e-5e42f15a438b-metrics-certs\") pod \"network-metrics-daemon-bzt4v\" (UID: \"6ab21338-9de8-4bef-a83e-5e42f15a438b\") " pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:54:10.956901 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.956872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2kpj\""
Sep 09 20:54:10.964597 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:10.964567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzt4v"
Sep 09 20:54:11.255534 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:11.255508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzt4v"]
Sep 09 20:54:11.257741 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:54:11.257694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab21338_9de8_4bef_a83e_5e42f15a438b.slice/crio-2e57e6bf1b1572cbf11305f6a5a6db118cb9306034809be9b7f88d90c54bc0ac WatchSource:0}: Error finding container 2e57e6bf1b1572cbf11305f6a5a6db118cb9306034809be9b7f88d90c54bc0ac: Status 404 returned error can't find the container with id 2e57e6bf1b1572cbf11305f6a5a6db118cb9306034809be9b7f88d90c54bc0ac
Sep 09 20:54:11.561837 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:11.561735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzt4v" event={"ID":"6ab21338-9de8-4bef-a83e-5e42f15a438b","Type":"ContainerStarted","Data":"2e57e6bf1b1572cbf11305f6a5a6db118cb9306034809be9b7f88d90c54bc0ac"}
Sep 09 20:54:11.563968 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:11.563935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gl642" event={"ID":"905f40ee-43b9-48ee-ab53-e6503d6aac84","Type":"ContainerStarted","Data":"be97452659b972501bea0e49573368fa14ddb1387104867ba37dc672d2dc8816"}
Sep 09 20:54:12.575769 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:12.575674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzt4v" event={"ID":"6ab21338-9de8-4bef-a83e-5e42f15a438b","Type":"ContainerStarted","Data":"8d2418773acb88b8b0c8739d7da6b9e4d98d65711ef9ab42a9f4fb155d0a225b"}
Sep 09 20:54:13.581796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:13.581758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzt4v" event={"ID":"6ab21338-9de8-4bef-a83e-5e42f15a438b","Type":"ContainerStarted","Data":"512abee18c885bb8f70cc6618c2f2170609d9ecc9c295bc17e42fea5450a62f6"}
Sep 09 20:54:13.598693 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:54:13.598637    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bzt4v" podStartSLOduration=130.458553311 podStartE2EDuration="2m11.598620558s" totalImagesPullingTime="1.140067247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:13.598276993 +0000 UTC m=+133.094551036" watchObservedRunningTime="2026-09-09 20:54:13.598620558 +0000 UTC m=+133.094894659"
Sep 09 20:55:04.917163 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:04.917080    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:55:29.942856 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:29.942790    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b89757cf4-vpdkh" podUID="affe4716-87f8-4306-91e5-b11d2828a72a" containerName="console" containerID="cri-o://076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40" gracePeriod=15
Sep 09 20:55:30.185443 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.185410    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:55:30.258256 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258256 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258256 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258533 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258533 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcbsv\" (UniqueName: \"kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258533 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258533 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca\") pod \"affe4716-87f8-4306-91e5-b11d2828a72a\" (UID: \"affe4716-87f8-4306-91e5-b11d2828a72a\") "
Sep 09 20:55:30.258776 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:30.258850 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config" (OuterVolumeSpecName: "console-config") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:30.258850 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:30.258929 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.258856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca" (OuterVolumeSpecName: "service-ca") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:30.260693 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.260660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv" (OuterVolumeSpecName: "kube-api-access-zcbsv") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "kube-api-access-zcbsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:30.260832 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.260719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:30.260832 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.260785    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "affe4716-87f8-4306-91e5-b11d2828a72a" (UID: "affe4716-87f8-4306-91e5-b11d2828a72a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:30.359089 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359049    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-serving-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359089 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359083    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-service-ca\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359089 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359092    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-trusted-ca-bundle\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359089 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359101    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affe4716-87f8-4306-91e5-b11d2828a72a-console-oauth-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359352 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359110    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-console-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359352 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359119    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affe4716-87f8-4306-91e5-b11d2828a72a-oauth-serving-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.359352 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.359129    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbsv\" (UniqueName: \"kubernetes.io/projected/affe4716-87f8-4306-91e5-b11d2828a72a-kube-api-access-zcbsv\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:55:30.856646 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.856609    2581 generic.go:356] "Generic (PLEG): container finished" podID="affe4716-87f8-4306-91e5-b11d2828a72a" containerID="076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40" exitCode=0
Sep 09 20:55:30.856646 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.856648    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b89757cf4-vpdkh"
Sep 09 20:55:30.856646 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.856652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b89757cf4-vpdkh" event={"ID":"affe4716-87f8-4306-91e5-b11d2828a72a","Type":"ContainerDied","Data":"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"}
Sep 09 20:55:30.856967 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.856725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b89757cf4-vpdkh" event={"ID":"affe4716-87f8-4306-91e5-b11d2828a72a","Type":"ContainerDied","Data":"f37968335f2050930049756cb99e8c50c6c26d6be36143ca1347b114317355fd"}
Sep 09 20:55:30.856967 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.856730    2581 scope.go:117] "RemoveContainer" containerID="076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"
Sep 09 20:55:30.865594 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.865567    2581 scope.go:117] "RemoveContainer" containerID="076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"
Sep 09 20:55:30.865929 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:55:30.865906    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40\": container with ID starting with 076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40 not found: ID does not exist" containerID="076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"
Sep 09 20:55:30.865995 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.865938    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40"} err="failed to get container status \"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40\": rpc error: code = NotFound desc = could not find container \"076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40\": container with ID starting with 076aae67a1422f19cda5d2e284b2ae35c527a2cd5e201a2858d77630499dcd40 not found: ID does not exist"
Sep 09 20:55:30.886193 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.886154    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:55:30.889398 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:30.889370    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b89757cf4-vpdkh"]
Sep 09 20:55:31.029994 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:55:31.029954    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="affe4716-87f8-4306-91e5-b11d2828a72a" path="/var/lib/kubelet/pods/affe4716-87f8-4306-91e5-b11d2828a72a/volumes"
Sep 09 20:56:50.576346 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.576311    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jtp9v"]
Sep 09 20:56:50.576796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.576574    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affe4716-87f8-4306-91e5-b11d2828a72a" containerName="console"
Sep 09 20:56:50.576796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.576588    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="affe4716-87f8-4306-91e5-b11d2828a72a" containerName="console"
Sep 09 20:56:50.576796 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.576635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="affe4716-87f8-4306-91e5-b11d2828a72a" containerName="console"
Sep 09 20:56:50.579444 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.579429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:50.590132 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.590110    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:56:50.590241 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.590132    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:56:50.590292 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.590255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:56:50.591401 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.591388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:56:50.591449 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.591390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zhkn4\""
Sep 09 20:56:50.635635 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.635604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:50.635799 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.635660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksw8\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-kube-api-access-2ksw8\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:50.636853 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.636832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jtp9v"]
Sep 09 20:56:50.736422 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.736381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksw8\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-kube-api-access-2ksw8\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:50.736630 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.736529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:50.736630 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:56:50.736613    2581 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 09 20:56:50.736630 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:56:50.736631    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-jtp9v: secret "keda-admission-webhooks-certs" not found
Sep 09 20:56:50.736835 ip-10-0-137-145 kubenswrapper[2581]: E0909 20:56:50.736689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates podName:5f3f5f31-e2be-409d-91a3-d808ebadf1ee nodeName:}" failed. No retries permitted until 2026-09-09 20:56:51.236671827 +0000 UTC m=+290.732945847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates") pod "keda-admission-6c4d864b86-jtp9v" (UID: "5f3f5f31-e2be-409d-91a3-d808ebadf1ee") : secret "keda-admission-webhooks-certs" not found
Sep 09 20:56:50.769987 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:50.769955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksw8\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-kube-api-access-2ksw8\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:51.240366 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:51.240321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:51.242899 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:51.242862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f3f5f31-e2be-409d-91a3-d808ebadf1ee-certificates\") pod \"keda-admission-6c4d864b86-jtp9v\" (UID: \"5f3f5f31-e2be-409d-91a3-d808ebadf1ee\") " pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:51.488826 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:51.488789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:51.772373 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:51.772296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jtp9v"]
Sep 09 20:56:51.774490 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:56:51.774461    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f5f31_e2be_409d_91a3_d808ebadf1ee.slice/crio-5ab4873ebaa84f5e4d1fba42a1326740c40f93ce20c60145a4472bce192b298a WatchSource:0}: Error finding container 5ab4873ebaa84f5e4d1fba42a1326740c40f93ce20c60145a4472bce192b298a: Status 404 returned error can't find the container with id 5ab4873ebaa84f5e4d1fba42a1326740c40f93ce20c60145a4472bce192b298a
Sep 09 20:56:52.124947 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:52.124864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v" event={"ID":"5f3f5f31-e2be-409d-91a3-d808ebadf1ee","Type":"ContainerStarted","Data":"5ab4873ebaa84f5e4d1fba42a1326740c40f93ce20c60145a4472bce192b298a"}
Sep 09 20:56:55.137424 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:55.137390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v" event={"ID":"5f3f5f31-e2be-409d-91a3-d808ebadf1ee","Type":"ContainerStarted","Data":"9be5c0d877e728b7080ec5a58545ccf962844884ea134dcbdc98880716e628d7"}
Sep 09 20:56:55.137847 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:55.137509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:56:55.183769 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:56:55.183716    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v" podStartSLOduration=2.6139828830000003 podStartE2EDuration="5.183683919s" totalImagesPullingTime="2.569701036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:55.182252834 +0000 UTC m=+294.678526855" watchObservedRunningTime="2026-09-09 20:56:55.183683919 +0000 UTC m=+294.679957960"
Sep 09 20:57:00.901446 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:57:00.901416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 20:57:00.904930 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:57:00.904906    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 20:57:00.911806 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:57:00.911766    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:57:16.143379 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:57:16.143344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jtp9v"
Sep 09 20:58:16.939561 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.939509    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:58:16.943082 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.943062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:16.951346 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.951322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:58:16.951457 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.951379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:58:16.952495 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.952479    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-xkczc\""
Sep 09 20:58:16.952586 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.952493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:58:16.981825 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:16.981672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:58:17.059870 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.059836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2c6\" (UniqueName: \"kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.060064 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.059999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.160724 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.160655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2c6\" (UniqueName: \"kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.160923 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.160756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.163357 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.163323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.190256 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.190186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2c6\" (UniqueName: \"kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6\") pod \"kserve-controller-manager-644949856f-lhnft\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") " pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.252129 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.252091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:17.530683 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.530602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:58:17.534048 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:58:17.534023    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d820624_bae7_4031_aad1_6b17a2c65e05.slice/crio-c9cdb01f35100046ee7780c0f7fb2f37bb373c313ee0873a1ce6e5089faadf71 WatchSource:0}: Error finding container c9cdb01f35100046ee7780c0f7fb2f37bb373c313ee0873a1ce6e5089faadf71: Status 404 returned error can't find the container with id c9cdb01f35100046ee7780c0f7fb2f37bb373c313ee0873a1ce6e5089faadf71
Sep 09 20:58:17.535387 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:17.535372    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:18.421590 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:18.421546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-lhnft" event={"ID":"0d820624-bae7-4031-aad1-6b17a2c65e05","Type":"ContainerStarted","Data":"c9cdb01f35100046ee7780c0f7fb2f37bb373c313ee0873a1ce6e5089faadf71"}
Sep 09 20:58:21.438994 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:21.438955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-lhnft" event={"ID":"0d820624-bae7-4031-aad1-6b17a2c65e05","Type":"ContainerStarted","Data":"8f37e961f118e960b13dcf7ee9f0226b91c930c4e5f5464e3908ca2f6434cba2"}
Sep 09 20:58:21.439414 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:21.439034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:58:21.469777 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:21.469723    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-lhnft" podStartSLOduration=2.525321075 podStartE2EDuration="5.469685068s" totalImagesPullingTime="2.944363993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:21.468945545 +0000 UTC m=+380.965219598" watchObservedRunningTime="2026-09-09 20:58:21.469685068 +0000 UTC m=+380.965959110"
Sep 09 20:58:29.259990 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.259953    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866f57cbb9-qvtl6"]
Sep 09 20:58:29.263136 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.263115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.267982 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.267954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:58:29.268099 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.267981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:58:29.268099 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.267993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:58:29.268215 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.268194    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:58:29.268268 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.268235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:58:29.268457 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.268422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:58:29.268582 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.268564    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:58:29.269972 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.269954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l62dd\""
Sep 09 20:58:29.272966 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.272946    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:58:29.280310 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.280289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866f57cbb9-qvtl6"]
Sep 09 20:58:29.358662 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358662 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-trusted-ca-bundle\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358905 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358905 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-oauth-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358905 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-service-ca\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358905 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgph\" (UniqueName: \"kubernetes.io/projected/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-kube-api-access-hcgph\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.358905 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.358859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-oauth-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460083 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460083 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-trusted-ca-bundle\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460285 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460285 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-oauth-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460285 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-service-ca\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460285 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgph\" (UniqueName: \"kubernetes.io/projected/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-kube-api-access-hcgph\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.460285 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-oauth-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.461012 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-service-ca\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.461139 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.460985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.461139 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.461026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-oauth-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.461390 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.461361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-trusted-ca-bundle\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.462777 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.462756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-oauth-config\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.462896 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.462861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-console-serving-cert\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.469505 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.469483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgph\" (UniqueName: \"kubernetes.io/projected/19f9a012-0e2b-4080-afe7-9b29e41e4d8f-kube-api-access-hcgph\") pod \"console-866f57cbb9-qvtl6\" (UID: \"19f9a012-0e2b-4080-afe7-9b29e41e4d8f\") " pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.574317 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.574229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:29.849806 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:29.849597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866f57cbb9-qvtl6"]
Sep 09 20:58:29.852333 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:58:29.852304    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f9a012_0e2b_4080_afe7_9b29e41e4d8f.slice/crio-6d8d7be66135afe9ed29bf7c889d3ac9617c9bbd5354302f5bed6dfa3b036a27 WatchSource:0}: Error finding container 6d8d7be66135afe9ed29bf7c889d3ac9617c9bbd5354302f5bed6dfa3b036a27: Status 404 returned error can't find the container with id 6d8d7be66135afe9ed29bf7c889d3ac9617c9bbd5354302f5bed6dfa3b036a27
Sep 09 20:58:30.472826 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:30.472790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866f57cbb9-qvtl6" event={"ID":"19f9a012-0e2b-4080-afe7-9b29e41e4d8f","Type":"ContainerStarted","Data":"0816853ccc151ed8555de597c23e15c37e015cb1ff03697702c3752e0c32b63f"}
Sep 09 20:58:30.473157 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:30.472845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866f57cbb9-qvtl6" event={"ID":"19f9a012-0e2b-4080-afe7-9b29e41e4d8f","Type":"ContainerStarted","Data":"6d8d7be66135afe9ed29bf7c889d3ac9617c9bbd5354302f5bed6dfa3b036a27"}
Sep 09 20:58:39.575195 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:39.575157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:39.575581 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:39.575266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:39.580606 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:39.580581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:39.604011 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:39.603960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866f57cbb9-qvtl6" podStartSLOduration=10.603943663 podStartE2EDuration="10.603943663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:30.523107318 +0000 UTC m=+390.019381379" watchObservedRunningTime="2026-09-09 20:58:39.603943663 +0000 UTC m=+399.100217704"
Sep 09 20:58:40.511169 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:40.511139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866f57cbb9-qvtl6"
Sep 09 20:58:52.447722 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:58:52.447669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:59:01.376017 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.375980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:59:01.376392 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.376175    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-644949856f-lhnft" podUID="0d820624-bae7-4031-aad1-6b17a2c65e05" containerName="manager" containerID="cri-o://8f37e961f118e960b13dcf7ee9f0226b91c930c4e5f5464e3908ca2f6434cba2" gracePeriod=10
Sep 09 20:59:01.398935 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.398906    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-6h6v2"]
Sep 09 20:59:01.401560 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.401543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.410442 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.410415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-6h6v2"]
Sep 09 20:59:01.414412 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.414383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqqh\" (UniqueName: \"kubernetes.io/projected/3f527382-9f9e-4d38-a115-609cff575578-kube-api-access-8zqqh\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.414540 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.414465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f527382-9f9e-4d38-a115-609cff575578-cert\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.515678 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.515641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f527382-9f9e-4d38-a115-609cff575578-cert\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.515852 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.515718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqqh\" (UniqueName: \"kubernetes.io/projected/3f527382-9f9e-4d38-a115-609cff575578-kube-api-access-8zqqh\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.518001 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.517941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f527382-9f9e-4d38-a115-609cff575578-cert\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.524018 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.523987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqqh\" (UniqueName: \"kubernetes.io/projected/3f527382-9f9e-4d38-a115-609cff575578-kube-api-access-8zqqh\") pod \"kserve-controller-manager-644949856f-6h6v2\" (UID: \"3f527382-9f9e-4d38-a115-609cff575578\") " pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.583811 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.583777    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d820624-bae7-4031-aad1-6b17a2c65e05" containerID="8f37e961f118e960b13dcf7ee9f0226b91c930c4e5f5464e3908ca2f6434cba2" exitCode=0
Sep 09 20:59:01.583961 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.583845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-lhnft" event={"ID":"0d820624-bae7-4031-aad1-6b17a2c65e05","Type":"ContainerDied","Data":"8f37e961f118e960b13dcf7ee9f0226b91c930c4e5f5464e3908ca2f6434cba2"}
Sep 09 20:59:01.618363 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.618337    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:59:01.717017 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.716979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2c6\" (UniqueName: \"kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6\") pod \"0d820624-bae7-4031-aad1-6b17a2c65e05\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") "
Sep 09 20:59:01.717196 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.717039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert\") pod \"0d820624-bae7-4031-aad1-6b17a2c65e05\" (UID: \"0d820624-bae7-4031-aad1-6b17a2c65e05\") "
Sep 09 20:59:01.719163 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.719129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert" (OuterVolumeSpecName: "cert") pod "0d820624-bae7-4031-aad1-6b17a2c65e05" (UID: "0d820624-bae7-4031-aad1-6b17a2c65e05"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:01.719163 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.719159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6" (OuterVolumeSpecName: "kube-api-access-mj2c6") pod "0d820624-bae7-4031-aad1-6b17a2c65e05" (UID: "0d820624-bae7-4031-aad1-6b17a2c65e05"). InnerVolumeSpecName "kube-api-access-mj2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:01.756396 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.756360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:01.818184 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.818155    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2c6\" (UniqueName: \"kubernetes.io/projected/0d820624-bae7-4031-aad1-6b17a2c65e05-kube-api-access-mj2c6\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:59:01.818184 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:01.818180    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d820624-bae7-4031-aad1-6b17a2c65e05-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 20:59:02.035966 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.035939    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-6h6v2"]
Sep 09 20:59:02.037926 ip-10-0-137-145 kubenswrapper[2581]: W0909 20:59:02.037902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f527382_9f9e_4d38_a115_609cff575578.slice/crio-29ab8b2e4bc07b738868ae357e234c1c3e765a2c0eea60d1bee6d3f8c8f071cb WatchSource:0}: Error finding container 29ab8b2e4bc07b738868ae357e234c1c3e765a2c0eea60d1bee6d3f8c8f071cb: Status 404 returned error can't find the container with id 29ab8b2e4bc07b738868ae357e234c1c3e765a2c0eea60d1bee6d3f8c8f071cb
Sep 09 20:59:02.588461 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.588433    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-lhnft"
Sep 09 20:59:02.588911 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.588431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-lhnft" event={"ID":"0d820624-bae7-4031-aad1-6b17a2c65e05","Type":"ContainerDied","Data":"c9cdb01f35100046ee7780c0f7fb2f37bb373c313ee0873a1ce6e5089faadf71"}
Sep 09 20:59:02.588911 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.588589    2581 scope.go:117] "RemoveContainer" containerID="8f37e961f118e960b13dcf7ee9f0226b91c930c4e5f5464e3908ca2f6434cba2"
Sep 09 20:59:02.590165 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.590137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-6h6v2" event={"ID":"3f527382-9f9e-4d38-a115-609cff575578","Type":"ContainerStarted","Data":"d7feaf9059cbb8fda2960fb9491b3f65ca636630d6111957324de30d29ba5d5f"}
Sep 09 20:59:02.590282 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.590178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-6h6v2" event={"ID":"3f527382-9f9e-4d38-a115-609cff575578","Type":"ContainerStarted","Data":"29ab8b2e4bc07b738868ae357e234c1c3e765a2c0eea60d1bee6d3f8c8f071cb"}
Sep 09 20:59:02.590282 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.590233    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 20:59:02.609085 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.609040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-6h6v2" podStartSLOduration=1.270152675 podStartE2EDuration="1.609025672s" totalImagesPullingTime="338.872997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:02.607886494 +0000 UTC m=+422.104160549" watchObservedRunningTime="2026-09-09 20:59:02.609025672 +0000 UTC m=+422.105299713"
Sep 09 20:59:02.623925 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.623892    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:59:02.632602 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:02.632535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-644949856f-lhnft"]
Sep 09 20:59:03.033737 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:03.030198    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d820624-bae7-4031-aad1-6b17a2c65e05" path="/var/lib/kubelet/pods/0d820624-bae7-4031-aad1-6b17a2c65e05/volumes"
Sep 09 20:59:33.600600 ip-10-0-137-145 kubenswrapper[2581]: I0909 20:59:33.600522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-6h6v2"
Sep 09 21:02:00.927338 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:02:00.927304    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:02:00.929036 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:02:00.929015    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:04:18.945781 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.945746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:04:18.946196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.946004    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d820624-bae7-4031-aad1-6b17a2c65e05" containerName="manager"
Sep 09 21:04:18.946196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.946015    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d820624-bae7-4031-aad1-6b17a2c65e05" containerName="manager"
Sep 09 21:04:18.946196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.946069    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d820624-bae7-4031-aad1-6b17a2c65e05" containerName="manager"
Sep 09 21:04:18.949520 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.949495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:18.954114 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:04:18.954262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 09 21:04:18.954262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 21:04:18.954354 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954160    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:04:18.954537 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:04:18.954949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.954934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:04:18.955348 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.955331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:04:18.967288 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:18.967255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:04:19.120938 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.120893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.120938 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.120941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.121143 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.120963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.121143 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.121054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.121143 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.121082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfj5\" (UniqueName: \"kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.221982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222336 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222336 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfj5\" (UniqueName: \"kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222469 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222720 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.222792 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.222769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.224589 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.224567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.232397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.232371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfj5\" (UniqueName: \"kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5\") pod \"isvc-predictive-xgboost-predictor-695f978978-jrc9x\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.260395 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.260360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:19.536018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.535980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:04:19.538320 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:04:19.538277    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ef1aa2_8460_41cc_ab6b_27840088e969.slice/crio-5acc65c30c37ffee6768764c3a6d1c46753002eae8d8d56eaf3f1369c8cac869 WatchSource:0}: Error finding container 5acc65c30c37ffee6768764c3a6d1c46753002eae8d8d56eaf3f1369c8cac869: Status 404 returned error can't find the container with id 5acc65c30c37ffee6768764c3a6d1c46753002eae8d8d56eaf3f1369c8cac869
Sep 09 21:04:19.540424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.540406    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:04:19.721678 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:19.721642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerStarted","Data":"5acc65c30c37ffee6768764c3a6d1c46753002eae8d8d56eaf3f1369c8cac869"}
Sep 09 21:04:23.739335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:23.739293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerStarted","Data":"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"}
Sep 09 21:04:27.756133 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:27.756096    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerID="a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738" exitCode=0
Sep 09 21:04:27.756499 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:27.756175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerDied","Data":"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"}
Sep 09 21:04:50.868598 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:50.868514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerStarted","Data":"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"}
Sep 09 21:04:53.883286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:53.883250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerStarted","Data":"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"}
Sep 09 21:04:53.883673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:53.883350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:53.903474 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:53.903416    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podStartSLOduration=2.929335451 podStartE2EDuration="35.903374786s" totalImagesPullingTime="29.103796268s" totalInitContainerRuntime="3.870243067s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:53.902646014 +0000 UTC m=+773.398920084" watchObservedRunningTime="2026-09-09 21:04:53.903374786 +0000 UTC m=+773.399648829"
Sep 09 21:04:54.887501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:54.887450    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:04:54.888721 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:54.888667    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:04:55.890547 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:04:55.890500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:00.894777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:00.894748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:05:00.895303 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:00.895277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:10.895432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:10.895391    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:20.895769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:20.895727    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:29.023797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.023713    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:05:29.026995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.026969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.031753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.031727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 09 21:05:29.032045 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.032028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 09 21:05:29.045333 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.045299    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:05:29.097723 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.097675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.097890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.097766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.097890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.097793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4d7\" (UniqueName: \"kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.198932 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.198894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.199127 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.198942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4d7\" (UniqueName: \"kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.199127 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.198994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.199777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.199754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.201365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.201345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.208608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.208578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4d7\" (UniqueName: \"kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7\") pod \"message-dumper-predictor-6575bcfdfd-f5fvc\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.337580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.337492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:29.612212 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:29.612121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:05:29.615520 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:05:29.615492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb55471_3383_49e8_adac_e66ab15f602d.slice/crio-94741526b7ce96571783275bc73173bbf3fa467ab4635c4900b314d5c14aa470 WatchSource:0}: Error finding container 94741526b7ce96571783275bc73173bbf3fa467ab4635c4900b314d5c14aa470: Status 404 returned error can't find the container with id 94741526b7ce96571783275bc73173bbf3fa467ab4635c4900b314d5c14aa470
Sep 09 21:05:30.026552 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:30.026519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerStarted","Data":"94741526b7ce96571783275bc73173bbf3fa467ab4635c4900b314d5c14aa470"}
Sep 09 21:05:30.896198 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:30.896162    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:31.033820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:31.033729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerStarted","Data":"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"}
Sep 09 21:05:31.033820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:31.033793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerStarted","Data":"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"}
Sep 09 21:05:31.034238 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:31.033851    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:31.052317 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:31.052251    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" podStartSLOduration=1.951639495 podStartE2EDuration="3.05222888s" totalImagesPullingTime="1.100589385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:31.051943212 +0000 UTC m=+810.548217273" watchObservedRunningTime="2026-09-09 21:05:31.05222888 +0000 UTC m=+810.548502927"
Sep 09 21:05:32.039343 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:32.039305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:32.043208 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:32.043182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:39.050160 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:39.050124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:05:40.895662 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:40.895616    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:05:50.895354 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:05:50.895316    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:06:00.895352 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:00.895311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 21:06:08.025864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:08.025831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:06:08.976190 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:08.976157    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:06:09.172452 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:09.172412    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container" containerID="cri-o://0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d" gracePeriod=30
Sep 09 21:06:09.172946 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:09.172483    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kube-rbac-proxy" containerID="cri-o://3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb" gracePeriod=30
Sep 09 21:06:10.178635 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:10.178597    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerID="3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb" exitCode=2
Sep 09 21:06:10.179013 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:10.178670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerDied","Data":"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"}
Sep 09 21:06:10.891210 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:10.891171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 09 21:06:13.902005 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:13.901980    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:06:14.064758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location\") pod \"f6ef1aa2-8460-41cc-ab6b-27840088e969\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") "
Sep 09 21:06:14.064758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qfj5\" (UniqueName: \"kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5\") pod \"f6ef1aa2-8460-41cc-ab6b-27840088e969\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") "
Sep 09 21:06:14.064758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config\") pod \"f6ef1aa2-8460-41cc-ab6b-27840088e969\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") "
Sep 09 21:06:14.065031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert\") pod \"f6ef1aa2-8460-41cc-ab6b-27840088e969\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") "
Sep 09 21:06:14.065031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls\") pod \"f6ef1aa2-8460-41cc-ab6b-27840088e969\" (UID: \"f6ef1aa2-8460-41cc-ab6b-27840088e969\") "
Sep 09 21:06:14.065031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.064973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6ef1aa2-8460-41cc-ab6b-27840088e969" (UID: "f6ef1aa2-8460-41cc-ab6b-27840088e969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:14.065192 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.065093    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6ef1aa2-8460-41cc-ab6b-27840088e969-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:06:14.065192 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.065157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6ef1aa2-8460-41cc-ab6b-27840088e969" (UID: "f6ef1aa2-8460-41cc-ab6b-27840088e969"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:14.065263 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.065206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6ef1aa2-8460-41cc-ab6b-27840088e969" (UID: "f6ef1aa2-8460-41cc-ab6b-27840088e969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:14.067187 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.067162    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5" (OuterVolumeSpecName: "kube-api-access-4qfj5") pod "f6ef1aa2-8460-41cc-ab6b-27840088e969" (UID: "f6ef1aa2-8460-41cc-ab6b-27840088e969"). InnerVolumeSpecName "kube-api-access-4qfj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:14.067311 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.067285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6ef1aa2-8460-41cc-ab6b-27840088e969" (UID: "f6ef1aa2-8460-41cc-ab6b-27840088e969"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:14.165724 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.165670    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6ef1aa2-8460-41cc-ab6b-27840088e969-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:06:14.165724 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.165722    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfj5\" (UniqueName: \"kubernetes.io/projected/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-api-access-4qfj5\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:06:14.165724 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.165732    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:06:14.165951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.165744    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6ef1aa2-8460-41cc-ab6b-27840088e969-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:06:14.195016 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.194981    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerID="0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d" exitCode=0
Sep 09 21:06:14.195168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.195037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerDied","Data":"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"}
Sep 09 21:06:14.195168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.195093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x" event={"ID":"f6ef1aa2-8460-41cc-ab6b-27840088e969","Type":"ContainerDied","Data":"5acc65c30c37ffee6768764c3a6d1c46753002eae8d8d56eaf3f1369c8cac869"}
Sep 09 21:06:14.195168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.195120    2581 scope.go:117] "RemoveContainer" containerID="3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"
Sep 09 21:06:14.195168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.195043    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"
Sep 09 21:06:14.204607 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.204585    2581 scope.go:117] "RemoveContainer" containerID="0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"
Sep 09 21:06:14.212444 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.212426    2581 scope.go:117] "RemoveContainer" containerID="a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"
Sep 09 21:06:14.220269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.220249    2581 scope.go:117] "RemoveContainer" containerID="3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"
Sep 09 21:06:14.220581 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:06:14.220562    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb\": container with ID starting with 3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb not found: ID does not exist" containerID="3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"
Sep 09 21:06:14.220629 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.220592    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb"} err="failed to get container status \"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb\": rpc error: code = NotFound desc = could not find container \"3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb\": container with ID starting with 3b67ff7b82108d9925830f57e78b4ddac5526860e19424a169c0a42319dac6bb not found: ID does not exist"
Sep 09 21:06:14.220629 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.220610    2581 scope.go:117] "RemoveContainer" containerID="0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"
Sep 09 21:06:14.220938 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:06:14.220915    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d\": container with ID starting with 0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d not found: ID does not exist" containerID="0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"
Sep 09 21:06:14.221035 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.220942    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d"} err="failed to get container status \"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d\": rpc error: code = NotFound desc = could not find container \"0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d\": container with ID starting with 0eb262278f73d3790e9449cba96383cc0c5390bb529ae98433339d2fbea4764d not found: ID does not exist"
Sep 09 21:06:14.221035 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.220958    2581 scope.go:117] "RemoveContainer" containerID="a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"
Sep 09 21:06:14.221208 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:06:14.221188    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738\": container with ID starting with a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738 not found: ID does not exist" containerID="a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"
Sep 09 21:06:14.221260 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.221217    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738"} err="failed to get container status \"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738\": rpc error: code = NotFound desc = could not find container \"a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738\": container with ID starting with a121fab7446e5b175b8b3e4a17a88b1f75e618a558d7980a0389222fa05ed738 not found: ID does not exist"
Sep 09 21:06:14.232657 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.232626    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:06:14.243336 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.243311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-jrc9x"]
Sep 09 21:06:14.303539 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303508    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:06:14.303825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303811    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kube-rbac-proxy"
Sep 09 21:06:14.303876 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kube-rbac-proxy"
Sep 09 21:06:14.303876 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303839    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="storage-initializer"
Sep 09 21:06:14.303876 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303850    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="storage-initializer"
Sep 09 21:06:14.303876 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303870    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container"
Sep 09 21:06:14.303876 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303876    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container"
Sep 09 21:06:14.304019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kserve-container"
Sep 09 21:06:14.304019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.303924    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" containerName="kube-rbac-proxy"
Sep 09 21:06:14.314404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.314360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.317940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.317879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 21:06:14.318187 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.318162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 09 21:06:14.318721 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.318692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:06:14.318721 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.318691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:06:14.329244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.329217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:06:14.467748 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.467691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.467934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.467757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.467934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.467863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nx6\" (UniqueName: \"kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.467934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.467900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.467934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.467925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.568753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.568651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nx6\" (UniqueName: \"kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.568753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.568693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.568753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.568747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.568990 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.568771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.568990 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.568793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.569207 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.569176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.569507 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.569486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.569579 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.569492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.571130 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.571111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.578334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.578307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nx6\" (UniqueName: \"kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.624234 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.624199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:14.895018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:14.894988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:06:14.896748 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:06:14.896721    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af0bd4_9ac5_4886_abb4_987c2d26d0b7.slice/crio-6218f0784a4db6237519006181abe18816eaa496ed895dc6042a1fedd15a3fa1 WatchSource:0}: Error finding container 6218f0784a4db6237519006181abe18816eaa496ed895dc6042a1fedd15a3fa1: Status 404 returned error can't find the container with id 6218f0784a4db6237519006181abe18816eaa496ed895dc6042a1fedd15a3fa1
Sep 09 21:06:15.030634 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:15.030601    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ef1aa2-8460-41cc-ab6b-27840088e969" path="/var/lib/kubelet/pods/f6ef1aa2-8460-41cc-ab6b-27840088e969/volumes"
Sep 09 21:06:15.203292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:15.203208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerStarted","Data":"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"}
Sep 09 21:06:15.203292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:15.203263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerStarted","Data":"6218f0784a4db6237519006181abe18816eaa496ed895dc6042a1fedd15a3fa1"}
Sep 09 21:06:19.220378 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:19.220338    2581 generic.go:356] "Generic (PLEG): container finished" podID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerID="5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61" exitCode=0
Sep 09 21:06:19.220866 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:19.220385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerDied","Data":"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"}
Sep 09 21:06:20.226493 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:20.226456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerStarted","Data":"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"}
Sep 09 21:06:20.226981 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:20.226562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerStarted","Data":"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"}
Sep 09 21:06:20.226981 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:20.226796    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:20.248072 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:20.247998    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podStartSLOduration=2.578351015 podStartE2EDuration="6.24798191s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.669630895s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:20.246720917 +0000 UTC m=+859.742994960" watchObservedRunningTime="2026-09-09 21:06:20.24798191 +0000 UTC m=+859.744255952"
Sep 09 21:06:21.230310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:21.230279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:21.231486 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:21.231461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:06:22.233540 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:22.233499    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:06:27.237422 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:27.237391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:06:27.237933 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:27.237904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:06:37.238311 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:37.238272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:06:47.238857 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:47.238814    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:06:57.237971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:06:57.237888    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:07:00.950822 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:00.950793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:07:00.951301 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:00.951103    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:07:07.238393 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:07.238349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:07:17.237898 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:17.237858    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:07:23.957204 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:23.957161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-f5fvc_bdb55471-3383-49e8-adac-e66ab15f602d/kserve-container/0.log"
Sep 09 21:07:24.170025 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.169991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:07:24.170266 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.170241    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kserve-container" containerID="cri-o://6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a" gracePeriod=30
Sep 09 21:07:24.170380 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.170331    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kube-rbac-proxy" containerID="cri-o://5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb" gracePeriod=30
Sep 09 21:07:24.282395 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.282321    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:07:24.285905 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.285884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.289041 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.289020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 09 21:07:24.289687 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.289673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 09 21:07:24.301839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.301808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:07:24.411891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.411866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgrf\" (UniqueName: \"kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.412027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.411937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.412027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.411977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.412027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.412016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.412162 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.412035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.412925 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.412904    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:07:24.466673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466632    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdb55471-3383-49e8-adac-e66ab15f602d" containerID="5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb" exitCode=2
Sep 09 21:07:24.466673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466662    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdb55471-3383-49e8-adac-e66ab15f602d" containerID="6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a" exitCode=2
Sep 09 21:07:24.466917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466711    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"
Sep 09 21:07:24.466917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerDied","Data":"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"}
Sep 09 21:07:24.466917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerDied","Data":"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"}
Sep 09 21:07:24.466917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc" event={"ID":"bdb55471-3383-49e8-adac-e66ab15f602d","Type":"ContainerDied","Data":"94741526b7ce96571783275bc73173bbf3fa467ab4635c4900b314d5c14aa470"}
Sep 09 21:07:24.466917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.466880    2581 scope.go:117] "RemoveContainer" containerID="5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"
Sep 09 21:07:24.474622 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.474600    2581 scope.go:117] "RemoveContainer" containerID="6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"
Sep 09 21:07:24.482106 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482090    2581 scope.go:117] "RemoveContainer" containerID="5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"
Sep 09 21:07:24.482372 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:07:24.482354    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb\": container with ID starting with 5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb not found: ID does not exist" containerID="5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"
Sep 09 21:07:24.482424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"} err="failed to get container status \"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb\": rpc error: code = NotFound desc = could not find container \"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb\": container with ID starting with 5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb not found: ID does not exist"
Sep 09 21:07:24.482424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482402    2581 scope.go:117] "RemoveContainer" containerID="6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"
Sep 09 21:07:24.482640 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:07:24.482621    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a\": container with ID starting with 6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a not found: ID does not exist" containerID="6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"
Sep 09 21:07:24.482681 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482647    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"} err="failed to get container status \"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a\": rpc error: code = NotFound desc = could not find container \"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a\": container with ID starting with 6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a not found: ID does not exist"
Sep 09 21:07:24.482681 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482664    2581 scope.go:117] "RemoveContainer" containerID="5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"
Sep 09 21:07:24.482939 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb"} err="failed to get container status \"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb\": rpc error: code = NotFound desc = could not find container \"5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb\": container with ID starting with 5b2815eb12f19b28b637b1681233db70bedc0c9e61c8a0f1c170cb6b413289fb not found: ID does not exist"
Sep 09 21:07:24.482975 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.482940    2581 scope.go:117] "RemoveContainer" containerID="6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"
Sep 09 21:07:24.483141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.483123    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a"} err="failed to get container status \"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a\": rpc error: code = NotFound desc = could not find container \"6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a\": container with ID starting with 6213018ae77228fb15626ffb0c9258dac3c1fd0e10f5b47deac9b4d1d5fcf83a not found: ID does not exist"
Sep 09 21:07:24.512753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.512726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config\") pod \"bdb55471-3383-49e8-adac-e66ab15f602d\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") "
Sep 09 21:07:24.512864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.512758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn4d7\" (UniqueName: \"kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7\") pod \"bdb55471-3383-49e8-adac-e66ab15f602d\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") "
Sep 09 21:07:24.512864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.512823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls\") pod \"bdb55471-3383-49e8-adac-e66ab15f602d\" (UID: \"bdb55471-3383-49e8-adac-e66ab15f602d\") "
Sep 09 21:07:24.512979 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.512960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgrf\" (UniqueName: \"kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bdb55471-3383-49e8-adac-e66ab15f602d" (UID: "bdb55471-3383-49e8-adac-e66ab15f602d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:24.513258 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513407 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513372    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdb55471-3383-49e8-adac-e66ab15f602d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.513452 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.513887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.513865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.514036 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.514016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.515415 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.515394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7" (OuterVolumeSpecName: "kube-api-access-mn4d7") pod "bdb55471-3383-49e8-adac-e66ab15f602d" (UID: "bdb55471-3383-49e8-adac-e66ab15f602d"). InnerVolumeSpecName "kube-api-access-mn4d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:24.515636 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.515615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.515636 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.515626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bdb55471-3383-49e8-adac-e66ab15f602d" (UID: "bdb55471-3383-49e8-adac-e66ab15f602d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:24.522162 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.522132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgrf\" (UniqueName: \"kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf\") pod \"isvc-autogluon-v1-predictor-574b5b7495-k7p4l\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.596348 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.596251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:07:24.613908 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.613882    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn4d7\" (UniqueName: \"kubernetes.io/projected/bdb55471-3383-49e8-adac-e66ab15f602d-kube-api-access-mn4d7\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.613995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.613910    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdb55471-3383-49e8-adac-e66ab15f602d-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.790817 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.790785    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:07:24.796587 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.796559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f5fvc"]
Sep 09 21:07:24.868978 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:24.868952    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:07:24.871410 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:07:24.871382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcf4fe3_3e5e_48a8_8c38_e70d454a01c5.slice/crio-efa4265ee1dac40b6577ab7da163a7b3a7989e39ea9f4ecfdf268dd071e7f238 WatchSource:0}: Error finding container efa4265ee1dac40b6577ab7da163a7b3a7989e39ea9f4ecfdf268dd071e7f238: Status 404 returned error can't find the container with id efa4265ee1dac40b6577ab7da163a7b3a7989e39ea9f4ecfdf268dd071e7f238
Sep 09 21:07:25.032031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:25.031991    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" path="/var/lib/kubelet/pods/bdb55471-3383-49e8-adac-e66ab15f602d/volumes"
Sep 09 21:07:25.472607 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:25.472569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerStarted","Data":"4899b55e610eefb5c111fbaf21f1f6b7eae557a664934de6343e48e54b34df9f"}
Sep 09 21:07:25.472801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:25.472642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerStarted","Data":"efa4265ee1dac40b6577ab7da163a7b3a7989e39ea9f4ecfdf268dd071e7f238"}
Sep 09 21:07:27.238249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:27.238204    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:07:33.024633 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:33.024575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 21:07:43.028783 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:43.028751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:07:44.223523 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:44.223486    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:07:44.223950 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:44.223742    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container" containerID="cri-o://cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a" gracePeriod=30
Sep 09 21:07:44.223950 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:44.223795    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kube-rbac-proxy" containerID="cri-o://9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58" gracePeriod=30
Sep 09 21:07:44.541951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:44.541869    2581 generic.go:356] "Generic (PLEG): container finished" podID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerID="9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58" exitCode=2
Sep 09 21:07:44.542089 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:44.541954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerDied","Data":"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"}
Sep 09 21:07:47.234024 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:47.233979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 21:07:49.177521 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.177489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:07:49.204781 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.204745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location\") pod \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") "
Sep 09 21:07:49.204959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.204791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls\") pod \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") "
Sep 09 21:07:49.204959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.204832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert\") pod \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") "
Sep 09 21:07:49.204959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.204850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9nx6\" (UniqueName: \"kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6\") pod \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") "
Sep 09 21:07:49.204959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.204894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config\") pod \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\" (UID: \"33af0bd4-9ac5-4886-abb4-987c2d26d0b7\") "
Sep 09 21:07:49.205269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.205233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33af0bd4-9ac5-4886-abb4-987c2d26d0b7" (UID: "33af0bd4-9ac5-4886-abb4-987c2d26d0b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:49.205379 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.205357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33af0bd4-9ac5-4886-abb4-987c2d26d0b7" (UID: "33af0bd4-9ac5-4886-abb4-987c2d26d0b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:49.205432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.205355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33af0bd4-9ac5-4886-abb4-987c2d26d0b7" (UID: "33af0bd4-9ac5-4886-abb4-987c2d26d0b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:49.206985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.206956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33af0bd4-9ac5-4886-abb4-987c2d26d0b7" (UID: "33af0bd4-9ac5-4886-abb4-987c2d26d0b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:49.207109 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.207054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6" (OuterVolumeSpecName: "kube-api-access-k9nx6") pod "33af0bd4-9ac5-4886-abb4-987c2d26d0b7" (UID: "33af0bd4-9ac5-4886-abb4-987c2d26d0b7"). InnerVolumeSpecName "kube-api-access-k9nx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:49.306175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.306091    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:49.306175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.306120    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:49.306175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.306131    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:49.306175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.306143    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:49.306175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.306152    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nx6\" (UniqueName: \"kubernetes.io/projected/33af0bd4-9ac5-4886-abb4-987c2d26d0b7-kube-api-access-k9nx6\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:07:49.561933 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.561845    2581 generic.go:356] "Generic (PLEG): container finished" podID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerID="cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a" exitCode=0
Sep 09 21:07:49.561933 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.561912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerDied","Data":"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"}
Sep 09 21:07:49.562113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.562003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4" event={"ID":"33af0bd4-9ac5-4886-abb4-987c2d26d0b7","Type":"ContainerDied","Data":"6218f0784a4db6237519006181abe18816eaa496ed895dc6042a1fedd15a3fa1"}
Sep 09 21:07:49.562113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.562019    2581 scope.go:117] "RemoveContainer" containerID="9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"
Sep 09 21:07:49.562113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.561927    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"
Sep 09 21:07:49.570525 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.570503    2581 scope.go:117] "RemoveContainer" containerID="cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"
Sep 09 21:07:49.579361 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.579338    2581 scope.go:117] "RemoveContainer" containerID="5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"
Sep 09 21:07:49.585550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.585522    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:07:49.587109 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.587092    2581 scope.go:117] "RemoveContainer" containerID="9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"
Sep 09 21:07:49.587437 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:07:49.587415    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58\": container with ID starting with 9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58 not found: ID does not exist" containerID="9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"
Sep 09 21:07:49.587477 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.587446    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58"} err="failed to get container status \"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58\": rpc error: code = NotFound desc = could not find container \"9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58\": container with ID starting with 9a5692a48eb9879f31213a3a43d4d83f2f250b24b8144bb585cb37ff92f58f58 not found: ID does not exist"
Sep 09 21:07:49.587477 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.587464    2581 scope.go:117] "RemoveContainer" containerID="cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"
Sep 09 21:07:49.587743 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:07:49.587722    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a\": container with ID starting with cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a not found: ID does not exist" containerID="cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"
Sep 09 21:07:49.587793 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.587752    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a"} err="failed to get container status \"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a\": rpc error: code = NotFound desc = could not find container \"cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a\": container with ID starting with cd4df2163d0de7d9b6a259058f68459b36126ff584bf40e60fb7e3c3fda7e04a not found: ID does not exist"
Sep 09 21:07:49.587793 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.587773    2581 scope.go:117] "RemoveContainer" containerID="5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"
Sep 09 21:07:49.588051 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:07:49.588032    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61\": container with ID starting with 5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61 not found: ID does not exist" containerID="5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"
Sep 09 21:07:49.588101 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.588055    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61"} err="failed to get container status \"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61\": rpc error: code = NotFound desc = could not find container \"5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61\": container with ID starting with 5dc1809e447315bc7f6110b94c655b8d31c8a5fb54b1de4715fb99474b708a61 not found: ID does not exist"
Sep 09 21:07:49.591717 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:49.591678    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-79pl4"]
Sep 09 21:07:51.029495 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:51.029462    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" path="/var/lib/kubelet/pods/33af0bd4-9ac5-4886-abb4-987c2d26d0b7/volumes"
Sep 09 21:07:54.582669 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:54.582634    2581 generic.go:356] "Generic (PLEG): container finished" podID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerID="4899b55e610eefb5c111fbaf21f1f6b7eae557a664934de6343e48e54b34df9f" exitCode=0
Sep 09 21:07:54.583069 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:07:54.582724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerDied","Data":"4899b55e610eefb5c111fbaf21f1f6b7eae557a664934de6343e48e54b34df9f"}
Sep 09 21:10:03.108949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.108859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerStarted","Data":"730aa9299f2d2a68496a33e418d9bce79de38d725c28b9932e7b34873c9093a2"}
Sep 09 21:10:03.108949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.108927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerStarted","Data":"9e46899920635a2cc531547a6442c69f7060ba645e165ee1a4cb1ddbb2e0587c"}
Sep 09 21:10:03.109550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.108962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:03.109550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.108989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:03.110498 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.110474    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 21:10:03.130231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:03.130178    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podStartSLOduration=2.360126112 podStartE2EDuration="2m39.130160771s" totalImagesPullingTime="2m7.857094876s" totalInitContainerRuntime="28.912939783s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:03.128820945 +0000 UTC m=+1082.625095015" watchObservedRunningTime="2026-09-09 21:10:03.130160771 +0000 UTC m=+1082.626434813"
Sep 09 21:10:04.113947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:04.113901    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 21:10:09.119493 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:09.119456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:09.120113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:09.120081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 09 21:10:19.120853 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:19.120824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:30.749421 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:30.749383    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:10:30.749874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:30.749614    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container" containerID="cri-o://9e46899920635a2cc531547a6442c69f7060ba645e165ee1a4cb1ddbb2e0587c" gracePeriod=30
Sep 09 21:10:30.749874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:30.749664    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kube-rbac-proxy" containerID="cri-o://730aa9299f2d2a68496a33e418d9bce79de38d725c28b9932e7b34873c9093a2" gracePeriod=30
Sep 09 21:10:31.218715 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:31.218673    2581 generic.go:356] "Generic (PLEG): container finished" podID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerID="730aa9299f2d2a68496a33e418d9bce79de38d725c28b9932e7b34873c9093a2" exitCode=2
Sep 09 21:10:31.218890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:31.218750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerDied","Data":"730aa9299f2d2a68496a33e418d9bce79de38d725c28b9932e7b34873c9093a2"}
Sep 09 21:10:32.225949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.225915    2581 generic.go:356] "Generic (PLEG): container finished" podID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerID="9e46899920635a2cc531547a6442c69f7060ba645e165ee1a4cb1ddbb2e0587c" exitCode=0
Sep 09 21:10:32.226305 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.225974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerDied","Data":"9e46899920635a2cc531547a6442c69f7060ba645e165ee1a4cb1ddbb2e0587c"}
Sep 09 21:10:32.318917 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.318876    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:32.492708 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.492672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcgrf\" (UniqueName: \"kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf\") pod \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") "
Sep 09 21:10:32.492930 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.492765    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location\") pod \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") "
Sep 09 21:10:32.492930 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.492811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config\") pod \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") "
Sep 09 21:10:32.492930 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.492827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert\") pod \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") "
Sep 09 21:10:32.492930 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.492854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls\") pod \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\" (UID: \"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5\") "
Sep 09 21:10:32.493234 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.493196    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" (UID: "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:32.493298 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.493250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" (UID: "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:32.495019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.494989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" (UID: "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:32.495136 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.495015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf" (OuterVolumeSpecName: "kube-api-access-wcgrf") pod "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" (UID: "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5"). InnerVolumeSpecName "kube-api-access-wcgrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:32.497722 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.497674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" (UID: "9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:32.593548 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.593510    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:10:32.593548 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.593544    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:10:32.593548 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.593556    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:10:32.593811 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.593564    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgrf\" (UniqueName: \"kubernetes.io/projected/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kube-api-access-wcgrf\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:10:32.593811 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:32.593573    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:10:33.231418 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.231389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"
Sep 09 21:10:33.231815 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.231392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l" event={"ID":"9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5","Type":"ContainerDied","Data":"efa4265ee1dac40b6577ab7da163a7b3a7989e39ea9f4ecfdf268dd071e7f238"}
Sep 09 21:10:33.231815 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.231541    2581 scope.go:117] "RemoveContainer" containerID="730aa9299f2d2a68496a33e418d9bce79de38d725c28b9932e7b34873c9093a2"
Sep 09 21:10:33.240197 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.240174    2581 scope.go:117] "RemoveContainer" containerID="9e46899920635a2cc531547a6442c69f7060ba645e165ee1a4cb1ddbb2e0587c"
Sep 09 21:10:33.247995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.247969    2581 scope.go:117] "RemoveContainer" containerID="4899b55e610eefb5c111fbaf21f1f6b7eae557a664934de6343e48e54b34df9f"
Sep 09 21:10:33.250070 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.250046    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:10:33.255933 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:33.255901    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-k7p4l"]
Sep 09 21:10:35.028841 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.028808    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" path="/var/lib/kubelet/pods/9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5/volumes"
Sep 09 21:10:35.952849 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.952817    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:10:35.953117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953118    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953146    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953152    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953159    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kserve-container"
Sep 09 21:10:35.953163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953165    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kserve-container"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953170    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="storage-initializer"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953175    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="storage-initializer"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953181    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953186    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953193    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="storage-initializer"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="storage-initializer"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953220    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953225    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953265    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kserve-container"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953273    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953278    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcf4fe3-3e5e-48a8-8c38-e70d454a01c5" containerName="kube-rbac-proxy"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953287    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb55471-3383-49e8-adac-e66ab15f602d" containerName="kserve-container"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953292    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kserve-container"
Sep 09 21:10:35.953417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.953297    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33af0bd4-9ac5-4886-abb4-987c2d26d0b7" containerName="kube-rbac-proxy"
Sep 09 21:10:35.989119 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.989076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:10:35.989283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.989215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:35.992136 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.992111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:10:35.992136 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.992126    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:10:35.992323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.992110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:10:35.993383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.993364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:10:35.993498 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.993415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:10:35.993498 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.993472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:10:35.993601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:35.993472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 09 21:10:36.124282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.124247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.124282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.124285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b46c\" (UniqueName: \"kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.124766 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.124378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.124766 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.124417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.124766 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.124454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225114 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225114 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b46c\" (UniqueName: \"kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225114 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225114 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225507 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225648 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225802 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.225851 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.225802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.227489 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.227467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.234142 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.234115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b46c\" (UniqueName: \"kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-h9zkq\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.299580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.299536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:10:36.592043 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:36.591961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:10:36.594857 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:10:36.594826    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d9b9c3_443e_474c_8c4e_8eacd50ca053.slice/crio-8132ea6bb49aa0dc3e2b1c913d341d534eaa8b0f3ba369960ca4b6c74ae37cec WatchSource:0}: Error finding container 8132ea6bb49aa0dc3e2b1c913d341d534eaa8b0f3ba369960ca4b6c74ae37cec: Status 404 returned error can't find the container with id 8132ea6bb49aa0dc3e2b1c913d341d534eaa8b0f3ba369960ca4b6c74ae37cec
Sep 09 21:10:37.249398 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:37.249353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerStarted","Data":"aa080cd30a9dd6e39052e56a12777dd35ca3e6bbcaa730d35b11ad6e5252e59e"}
Sep 09 21:10:37.249825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:10:37.249444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerStarted","Data":"8132ea6bb49aa0dc3e2b1c913d341d534eaa8b0f3ba369960ca4b6c74ae37cec"}
Sep 09 21:11:06.354228 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:06.354194    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerID="aa080cd30a9dd6e39052e56a12777dd35ca3e6bbcaa730d35b11ad6e5252e59e" exitCode=0
Sep 09 21:11:06.354670 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:06.354275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerDied","Data":"aa080cd30a9dd6e39052e56a12777dd35ca3e6bbcaa730d35b11ad6e5252e59e"}
Sep 09 21:11:07.360832 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:07.360799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerStarted","Data":"2f3c318e102c73f51ad387d8160c6d3c06acbe97053a60ea5273b0cf05ad0816"}
Sep 09 21:11:07.361216 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:07.360863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerStarted","Data":"2b1740c5a65d976d2c76347bd41b87535d2d2431a9d81f91bdbf987533c6deb4"}
Sep 09 21:11:07.361216 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:07.361071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:11:07.381889 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:07.381835    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" podStartSLOduration=3.748840221 podStartE2EDuration="32.381818466s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.632978245s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:07.380585808 +0000 UTC m=+1146.876859870" watchObservedRunningTime="2026-09-09 21:11:07.381818466 +0000 UTC m=+1146.878092508"
Sep 09 21:11:08.366451 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:08.366416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:11:14.376295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:11:14.376266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:12:00.973924 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:00.973894    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:12:00.974462 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:00.974388    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:12:39.030350 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:39.030314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:12:50.067895 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:50.067838    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:12:52.302245 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:52.302209    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:12:52.302638 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:52.302499    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kserve-container" containerID="cri-o://2b1740c5a65d976d2c76347bd41b87535d2d2431a9d81f91bdbf987533c6deb4" gracePeriod=30
Sep 09 21:12:52.302638 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:52.302517    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kube-rbac-proxy" containerID="cri-o://2f3c318e102c73f51ad387d8160c6d3c06acbe97053a60ea5273b0cf05ad0816" gracePeriod=30
Sep 09 21:12:52.759233 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:52.759192    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerID="2f3c318e102c73f51ad387d8160c6d3c06acbe97053a60ea5273b0cf05ad0816" exitCode=2
Sep 09 21:12:52.759431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:52.759262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerDied","Data":"2f3c318e102c73f51ad387d8160c6d3c06acbe97053a60ea5273b0cf05ad0816"}
Sep 09 21:12:53.765186 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:53.765154    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerID="2b1740c5a65d976d2c76347bd41b87535d2d2431a9d81f91bdbf987533c6deb4" exitCode=0
Sep 09 21:12:53.765550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:53.765232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerDied","Data":"2b1740c5a65d976d2c76347bd41b87535d2d2431a9d81f91bdbf987533c6deb4"}
Sep 09 21:12:53.851390 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:53.851362    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:12:54.000816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.000720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls\") pod \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") "
Sep 09 21:12:54.000816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.000783    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config\") pod \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") "
Sep 09 21:12:54.000816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.000810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location\") pod \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") "
Sep 09 21:12:54.001036 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.000907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert\") pod \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") "
Sep 09 21:12:54.001036 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.000945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b46c\" (UniqueName: \"kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c\") pod \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\" (UID: \"e8d9b9c3-443e-474c-8c4e-8eacd50ca053\") "
Sep 09 21:12:54.001188 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.001163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8d9b9c3-443e-474c-8c4e-8eacd50ca053" (UID: "e8d9b9c3-443e-474c-8c4e-8eacd50ca053"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:54.001269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.001245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8d9b9c3-443e-474c-8c4e-8eacd50ca053" (UID: "e8d9b9c3-443e-474c-8c4e-8eacd50ca053"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:54.002995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.002963    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8d9b9c3-443e-474c-8c4e-8eacd50ca053" (UID: "e8d9b9c3-443e-474c-8c4e-8eacd50ca053"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:54.003104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.003038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c" (OuterVolumeSpecName: "kube-api-access-4b46c") pod "e8d9b9c3-443e-474c-8c4e-8eacd50ca053" (UID: "e8d9b9c3-443e-474c-8c4e-8eacd50ca053"). InnerVolumeSpecName "kube-api-access-4b46c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:54.005440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.005417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8d9b9c3-443e-474c-8c4e-8eacd50ca053" (UID: "e8d9b9c3-443e-474c-8c4e-8eacd50ca053"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:54.101672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.101621    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.101672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.101667    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.101672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.101678    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.101672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.101687    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.101981 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.101715    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b46c\" (UniqueName: \"kubernetes.io/projected/e8d9b9c3-443e-474c-8c4e-8eacd50ca053-kube-api-access-4b46c\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:12:54.771153 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.771111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq" event={"ID":"e8d9b9c3-443e-474c-8c4e-8eacd50ca053","Type":"ContainerDied","Data":"8132ea6bb49aa0dc3e2b1c913d341d534eaa8b0f3ba369960ca4b6c74ae37cec"}
Sep 09 21:12:54.771153 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.771133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"
Sep 09 21:12:54.771637 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.771179    2581 scope.go:117] "RemoveContainer" containerID="2f3c318e102c73f51ad387d8160c6d3c06acbe97053a60ea5273b0cf05ad0816"
Sep 09 21:12:54.779587 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.779565    2581 scope.go:117] "RemoveContainer" containerID="2b1740c5a65d976d2c76347bd41b87535d2d2431a9d81f91bdbf987533c6deb4"
Sep 09 21:12:54.787460 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.787435    2581 scope.go:117] "RemoveContainer" containerID="aa080cd30a9dd6e39052e56a12777dd35ca3e6bbcaa730d35b11ad6e5252e59e"
Sep 09 21:12:54.793489 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.793458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:12:54.798601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:54.798576    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-h9zkq"]
Sep 09 21:12:55.029528 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:55.029432    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" path="/var/lib/kubelet/pods/e8d9b9c3-443e-474c-8c4e-8eacd50ca053/volumes"
Sep 09 21:12:57.480337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480296    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kube-rbac-proxy"
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kube-rbac-proxy"
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="storage-initializer"
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480755    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="storage-initializer"
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kserve-container"
Sep 09 21:12:57.480831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480773    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kserve-container"
Sep 09 21:12:57.481141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480850    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kube-rbac-proxy"
Sep 09 21:12:57.481141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.480861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d9b9c3-443e-474c-8c4e-8eacd50ca053" containerName="kserve-container"
Sep 09 21:12:57.485674 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.485651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.488211 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.488188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 09 21:12:57.488433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.488413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:12:57.488512 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.488430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:12:57.489453 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.489431    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 09 21:12:57.489538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.489503    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:12:57.489538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.489435    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:12:57.489661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.489548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:12:57.497976 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.497948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:12:57.626994 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.626955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.626994 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.627002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.627229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.627033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.627229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.627055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x544w\" (UniqueName: \"kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.627229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.627083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.727763 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.727720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.727998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.727778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.727998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.727827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.727998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.727871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x544w\" (UniqueName: \"kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.727998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.727893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.728374 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.728344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.728579 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.728561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.728622 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.728562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.730287 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.730262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.737631 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.737565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x544w\" (UniqueName: \"kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:57.795879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:57.795838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:12:58.068434 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:58.068356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:12:58.071527 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:12:58.071498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52fb34f_675c_4fd0_9a3a_d3a858d83350.slice/crio-7377ab5052ba7b6a34b09fbb0e5125e45d588f95d322771a032922845ec869bc WatchSource:0}: Error finding container 7377ab5052ba7b6a34b09fbb0e5125e45d588f95d322771a032922845ec869bc: Status 404 returned error can't find the container with id 7377ab5052ba7b6a34b09fbb0e5125e45d588f95d322771a032922845ec869bc
Sep 09 21:12:58.788598 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:58.788555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerStarted","Data":"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"}
Sep 09 21:12:58.789050 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:12:58.788631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerStarted","Data":"7377ab5052ba7b6a34b09fbb0e5125e45d588f95d322771a032922845ec869bc"}
Sep 09 21:13:03.808990 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:03.808952    2581 generic.go:356] "Generic (PLEG): container finished" podID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerID="d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e" exitCode=0
Sep 09 21:13:03.809399 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:03.809025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerDied","Data":"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"}
Sep 09 21:13:04.815243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:04.815204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerStarted","Data":"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"}
Sep 09 21:13:04.815726 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:04.815276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerStarted","Data":"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"}
Sep 09 21:13:04.815726 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:04.815509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:13:04.815726 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:04.815567    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:13:04.834213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:04.834158    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" podStartSLOduration=2.730128016 podStartE2EDuration="7.834139588s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.104011572s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:04.833404269 +0000 UTC m=+1264.329678338" watchObservedRunningTime="2026-09-09 21:13:04.834139588 +0000 UTC m=+1264.330413630"
Sep 09 21:13:10.824431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:13:10.824404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:14:33.030094 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:33.030021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:14:44.331636 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:44.331598    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:14:44.332061 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:44.331902    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kube-rbac-proxy" containerID="cri-o://5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb" gracePeriod=30
Sep 09 21:14:44.332061 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:44.331886    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kserve-container" containerID="cri-o://cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d" gracePeriod=30
Sep 09 21:14:45.190167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.190133    2581 generic.go:356] "Generic (PLEG): container finished" podID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerID="5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb" exitCode=2
Sep 09 21:14:45.190372 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.190192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerDied","Data":"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"}
Sep 09 21:14:45.875548 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.875521    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:14:45.966986 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.966894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls\") pod \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") "
Sep 09 21:14:45.966986 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.966951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config\") pod \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") "
Sep 09 21:14:45.966986 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.966982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert\") pod \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") "
Sep 09 21:14:45.967224 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.967001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x544w\" (UniqueName: \"kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w\") pod \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") "
Sep 09 21:14:45.967224 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.967029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location\") pod \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\" (UID: \"d52fb34f-675c-4fd0-9a3a-d3a858d83350\") "
Sep 09 21:14:45.967384 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.967364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d52fb34f-675c-4fd0-9a3a-d3a858d83350" (UID: "d52fb34f-675c-4fd0-9a3a-d3a858d83350"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:45.967451 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.967393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d52fb34f-675c-4fd0-9a3a-d3a858d83350" (UID: "d52fb34f-675c-4fd0-9a3a-d3a858d83350"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:45.969161 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.969134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d52fb34f-675c-4fd0-9a3a-d3a858d83350" (UID: "d52fb34f-675c-4fd0-9a3a-d3a858d83350"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:45.969295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.969243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w" (OuterVolumeSpecName: "kube-api-access-x544w") pod "d52fb34f-675c-4fd0-9a3a-d3a858d83350" (UID: "d52fb34f-675c-4fd0-9a3a-d3a858d83350"). InnerVolumeSpecName "kube-api-access-x544w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:45.971693 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:45.971666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d52fb34f-675c-4fd0-9a3a-d3a858d83350" (UID: "d52fb34f-675c-4fd0-9a3a-d3a858d83350"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:46.067568 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.067528    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52fb34f-675c-4fd0-9a3a-d3a858d83350-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.067568 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.067561    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.067568 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.067571    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d52fb34f-675c-4fd0-9a3a-d3a858d83350-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.067827 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.067581    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x544w\" (UniqueName: \"kubernetes.io/projected/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kube-api-access-x544w\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.067827 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.067590    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d52fb34f-675c-4fd0-9a3a-d3a858d83350-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.196099 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.196058    2581 generic.go:356] "Generic (PLEG): container finished" podID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerID="cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d" exitCode=0
Sep 09 21:14:46.196297 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.196117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"
Sep 09 21:14:46.196297 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.196139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerDied","Data":"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"}
Sep 09 21:14:46.196297 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.196212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" event={"ID":"d52fb34f-675c-4fd0-9a3a-d3a858d83350","Type":"ContainerDied","Data":"7377ab5052ba7b6a34b09fbb0e5125e45d588f95d322771a032922845ec869bc"}
Sep 09 21:14:46.196297 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.196252    2581 scope.go:117] "RemoveContainer" containerID="5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"
Sep 09 21:14:46.204593 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.204575    2581 scope.go:117] "RemoveContainer" containerID="cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"
Sep 09 21:14:46.212497 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.212478    2581 scope.go:117] "RemoveContainer" containerID="d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"
Sep 09 21:14:46.218454 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.218380    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:14:46.220482 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.220463    2581 scope.go:117] "RemoveContainer" containerID="5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"
Sep 09 21:14:46.220843 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:14:46.220822    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb\": container with ID starting with 5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb not found: ID does not exist" containerID="5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"
Sep 09 21:14:46.220908 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.220859    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb"} err="failed to get container status \"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb\": rpc error: code = NotFound desc = could not find container \"5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb\": container with ID starting with 5451e7ac7292c539218fb06adb256084d5f48f368527a69f830ecff899a02ecb not found: ID does not exist"
Sep 09 21:14:46.220908 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.220887    2581 scope.go:117] "RemoveContainer" containerID="cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"
Sep 09 21:14:46.221192 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:14:46.221172    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d\": container with ID starting with cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d not found: ID does not exist" containerID="cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"
Sep 09 21:14:46.221246 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.221200    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d"} err="failed to get container status \"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d\": rpc error: code = NotFound desc = could not find container \"cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d\": container with ID starting with cdaa6f9371b91f92390396e98ca2cd281566110a96e3ed3ee3f37e652a6dc74d not found: ID does not exist"
Sep 09 21:14:46.221246 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.221225    2581 scope.go:117] "RemoveContainer" containerID="d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"
Sep 09 21:14:46.221480 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:14:46.221462    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e\": container with ID starting with d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e not found: ID does not exist" containerID="d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"
Sep 09 21:14:46.221565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.221546    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e"} err="failed to get container status \"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e\": rpc error: code = NotFound desc = could not find container \"d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e\": container with ID starting with d5182cebc8b0ad38c093a0057e90a3450c30ecfe0594ad297f7c51230284ad0e not found: ID does not exist"
Sep 09 21:14:46.223415 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.223384    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f"]
Sep 09 21:14:46.819649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:46.819604    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7r24f" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:14:47.034757 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:47.034713    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" path="/var/lib/kubelet/pods/d52fb34f-675c-4fd0-9a3a-d3a858d83350/volumes"
Sep 09 21:14:49.569139 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kserve-container"
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kserve-container"
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569539    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kube-rbac-proxy"
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569547    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kube-rbac-proxy"
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="storage-initializer"
Sep 09 21:14:49.569617 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569570    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="storage-initializer"
Sep 09 21:14:49.569942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kserve-container"
Sep 09 21:14:49.569942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.569648    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52fb34f-675c-4fd0-9a3a-d3a858d83350" containerName="kube-rbac-proxy"
Sep 09 21:14:49.574514 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.574486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.577517 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.577490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:14:49.577680 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.577545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:14:49.578390 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.578364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:14:49.578515 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.578383    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:14:49.578675 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.578659    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 09 21:14:49.578761 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.578746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 09 21:14:49.578847 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.578831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:14:49.588312 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.588286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:14:49.693878 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.693835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.693878 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.693881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.694091 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.693952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.694091 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.694022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.694216 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.694099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7km6\" (UniqueName: \"kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.794610 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.794564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.794823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.794616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.794823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.794689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7km6\" (UniqueName: \"kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.794823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.794741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.794823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.794769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.795096 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.795072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.795328 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.795306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.795397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.795344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.797146 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.797126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.804497 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.804467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7km6\" (UniqueName: \"kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:49.885859 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:49.885774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:50.163266 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:50.163097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:14:50.166043 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:14:50.166017    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode141bc96_d732_426f_ac08_57d98b602098.slice/crio-4fb7824494d7229a41494bf2f0d2681e4a07909c9fc8c316be44dac8e9017275 WatchSource:0}: Error finding container 4fb7824494d7229a41494bf2f0d2681e4a07909c9fc8c316be44dac8e9017275: Status 404 returned error can't find the container with id 4fb7824494d7229a41494bf2f0d2681e4a07909c9fc8c316be44dac8e9017275
Sep 09 21:14:50.213351 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:50.213320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerStarted","Data":"4fb7824494d7229a41494bf2f0d2681e4a07909c9fc8c316be44dac8e9017275"}
Sep 09 21:14:51.218790 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:51.218750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerStarted","Data":"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"}
Sep 09 21:14:55.236371 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:55.236334    2581 generic.go:356] "Generic (PLEG): container finished" podID="e141bc96-d732-426f-ac08-57d98b602098" containerID="ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba" exitCode=0
Sep 09 21:14:55.236828 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:55.236410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerDied","Data":"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"}
Sep 09 21:14:56.243159 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:56.243124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerStarted","Data":"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"}
Sep 09 21:14:56.243571 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:56.243189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerStarted","Data":"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"}
Sep 09 21:14:56.243571 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:56.243481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:56.243571 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:56.243535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:14:56.263326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:14:56.263274    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" podStartSLOduration=2.359406538 podStartE2EDuration="7.263257457s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.903850919s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:56.262003411 +0000 UTC m=+1375.758277472" watchObservedRunningTime="2026-09-09 21:14:56.263257457 +0000 UTC m=+1375.759531499"
Sep 09 21:15:02.254459 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:15:02.254426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:16:31.031968 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:31.031936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:16:42.068883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:42.068840    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:16:45.883782 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:45.883746    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:16:45.884226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:45.883994    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kserve-container" containerID="cri-o://430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81" gracePeriod=30
Sep 09 21:16:45.884226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:45.884029    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kube-rbac-proxy" containerID="cri-o://39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff" gracePeriod=30
Sep 09 21:16:46.650284 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:46.650248    2581 generic.go:356] "Generic (PLEG): container finished" podID="e141bc96-d732-426f-ac08-57d98b602098" containerID="39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff" exitCode=2
Sep 09 21:16:46.650457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:46.650324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerDied","Data":"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"}
Sep 09 21:16:47.249039 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.248991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 09 21:16:47.326973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.326947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:16:47.514581 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location\") pod \"e141bc96-d732-426f-ac08-57d98b602098\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") "
Sep 09 21:16:47.514787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514594    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls\") pod \"e141bc96-d732-426f-ac08-57d98b602098\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") "
Sep 09 21:16:47.514787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config\") pod \"e141bc96-d732-426f-ac08-57d98b602098\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") "
Sep 09 21:16:47.514787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7km6\" (UniqueName: \"kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6\") pod \"e141bc96-d732-426f-ac08-57d98b602098\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") "
Sep 09 21:16:47.514957 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert\") pod \"e141bc96-d732-426f-ac08-57d98b602098\" (UID: \"e141bc96-d732-426f-ac08-57d98b602098\") "
Sep 09 21:16:47.515100 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.514997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e141bc96-d732-426f-ac08-57d98b602098" (UID: "e141bc96-d732-426f-ac08-57d98b602098"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:47.515213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.515186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e141bc96-d732-426f-ac08-57d98b602098" (UID: "e141bc96-d732-426f-ac08-57d98b602098"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:47.516756 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.516726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6" (OuterVolumeSpecName: "kube-api-access-w7km6") pod "e141bc96-d732-426f-ac08-57d98b602098" (UID: "e141bc96-d732-426f-ac08-57d98b602098"). InnerVolumeSpecName "kube-api-access-w7km6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:47.517082 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.517058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e141bc96-d732-426f-ac08-57d98b602098" (UID: "e141bc96-d732-426f-ac08-57d98b602098"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:47.519084 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.519060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e141bc96-d732-426f-ac08-57d98b602098" (UID: "e141bc96-d732-426f-ac08-57d98b602098"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:47.615366 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.615324    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7km6\" (UniqueName: \"kubernetes.io/projected/e141bc96-d732-426f-ac08-57d98b602098-kube-api-access-w7km6\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:16:47.615366 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.615364    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:16:47.615366 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.615375    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e141bc96-d732-426f-ac08-57d98b602098-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:16:47.615609 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.615384    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e141bc96-d732-426f-ac08-57d98b602098-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:16:47.615609 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.615393    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e141bc96-d732-426f-ac08-57d98b602098-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:16:47.655847 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.655811    2581 generic.go:356] "Generic (PLEG): container finished" podID="e141bc96-d732-426f-ac08-57d98b602098" containerID="430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81" exitCode=0
Sep 09 21:16:47.656021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.655874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"
Sep 09 21:16:47.656021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.655895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerDied","Data":"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"}
Sep 09 21:16:47.656021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.655956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb" event={"ID":"e141bc96-d732-426f-ac08-57d98b602098","Type":"ContainerDied","Data":"4fb7824494d7229a41494bf2f0d2681e4a07909c9fc8c316be44dac8e9017275"}
Sep 09 21:16:47.656021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.655984    2581 scope.go:117] "RemoveContainer" containerID="39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"
Sep 09 21:16:47.664543 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.664525    2581 scope.go:117] "RemoveContainer" containerID="430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"
Sep 09 21:16:47.672072 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.672048    2581 scope.go:117] "RemoveContainer" containerID="ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"
Sep 09 21:16:47.679484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.679460    2581 scope.go:117] "RemoveContainer" containerID="39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"
Sep 09 21:16:47.679929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.679909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:16:47.679996 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:16:47.679957    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff\": container with ID starting with 39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff not found: ID does not exist" containerID="39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"
Sep 09 21:16:47.679996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.679985    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff"} err="failed to get container status \"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff\": rpc error: code = NotFound desc = could not find container \"39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff\": container with ID starting with 39a61be9ac03748abf60c581039326e64d6a0d0c1f784a74f7be2bec58a0fbff not found: ID does not exist"
Sep 09 21:16:47.680076 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.680003    2581 scope.go:117] "RemoveContainer" containerID="430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"
Sep 09 21:16:47.680294 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:16:47.680276    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81\": container with ID starting with 430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81 not found: ID does not exist" containerID="430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"
Sep 09 21:16:47.680343 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.680300    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81"} err="failed to get container status \"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81\": rpc error: code = NotFound desc = could not find container \"430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81\": container with ID starting with 430603f983e40c3e1f062d447631b732f855b47cf7a2e494dc99cd2fbbe7ac81 not found: ID does not exist"
Sep 09 21:16:47.680343 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.680316    2581 scope.go:117] "RemoveContainer" containerID="ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"
Sep 09 21:16:47.680572 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:16:47.680555    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba\": container with ID starting with ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba not found: ID does not exist" containerID="ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"
Sep 09 21:16:47.680610 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.680577    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba"} err="failed to get container status \"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba\": rpc error: code = NotFound desc = could not find container \"ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba\": container with ID starting with ee04bf55f7d688de43008c876fa691eb32ae4e868ed09a2f7983e1625e5595ba not found: ID does not exist"
Sep 09 21:16:47.683890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:47.683870    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gjvcb"]
Sep 09 21:16:49.029190 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:49.029158    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e141bc96-d732-426f-ac08-57d98b602098" path="/var/lib/kubelet/pods/e141bc96-d732-426f-ac08-57d98b602098/volumes"
Sep 09 21:16:51.152794 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.152755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153138    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kube-rbac-proxy"
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153156    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kube-rbac-proxy"
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153169    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="storage-initializer"
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153178    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="storage-initializer"
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153194    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kserve-container"
Sep 09 21:16:51.153239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153202    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kserve-container"
Sep 09 21:16:51.153554 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153317    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kserve-container"
Sep 09 21:16:51.153554 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.153332    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e141bc96-d732-426f-ac08-57d98b602098" containerName="kube-rbac-proxy"
Sep 09 21:16:51.158003 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.157979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.161843 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.161812    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:16:51.161843 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.161837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 09 21:16:51.162024 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.161878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:16:51.162118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.161812    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:16:51.162256 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.162238    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:16:51.162882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.162834    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:16:51.162882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.162851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 09 21:16:51.170599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.170574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:16:51.242678 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.242643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.242678 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.242683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.242900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.242719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm79l\" (UniqueName: \"kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.242900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.242755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.242900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.242792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.343516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.343485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.343516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.343523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.343812 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.343548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm79l\" (UniqueName: \"kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.343873 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.343814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.343929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.343878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.344218 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.344188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.344310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.344259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.344310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.344277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.346294 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.346276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.353982 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.353960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm79l\" (UniqueName: \"kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.471318 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.471222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:16:51.752430 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:51.752400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:16:51.753984 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:16:51.753960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a657e8e_5745_406d_8385_53cfb56def82.slice/crio-6a0e5c915d4cfc493d9687a3b67ffb80eb1f0230681e488e1ec26c3fdc84717b WatchSource:0}: Error finding container 6a0e5c915d4cfc493d9687a3b67ffb80eb1f0230681e488e1ec26c3fdc84717b: Status 404 returned error can't find the container with id 6a0e5c915d4cfc493d9687a3b67ffb80eb1f0230681e488e1ec26c3fdc84717b
Sep 09 21:16:52.677271 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:52.677236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerStarted","Data":"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"}
Sep 09 21:16:52.677618 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:16:52.677305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerStarted","Data":"6a0e5c915d4cfc493d9687a3b67ffb80eb1f0230681e488e1ec26c3fdc84717b"}
Sep 09 21:17:00.994657 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:00.994630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:17:00.996546 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:00.996522    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:17:01.709916 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:01.709879    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a657e8e-5745-406d-8385-53cfb56def82" containerID="4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4" exitCode=0
Sep 09 21:17:01.710087 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:01.709954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerDied","Data":"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"}
Sep 09 21:17:02.716390 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:02.716352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerStarted","Data":"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"}
Sep 09 21:17:02.716882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:02.716442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerStarted","Data":"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"}
Sep 09 21:17:02.716882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:02.716659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:02.737126 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:02.737074    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podStartSLOduration=2.317711106 podStartE2EDuration="11.737058789s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.419347683s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:17:02.73651526 +0000 UTC m=+1502.232789302" watchObservedRunningTime="2026-09-09 21:17:02.737058789 +0000 UTC m=+1502.233332830"
Sep 09 21:17:03.719777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:03.719744    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:03.721039 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:03.721000    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 21:17:04.723861 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:04.723814    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 21:17:09.729218 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:09.729179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:09.730287 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:09.730249    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 21:17:19.730985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:19.730949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:33.674165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:33.674080    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:17:33.674565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:33.674347    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container" containerID="cri-o://cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839" gracePeriod=30
Sep 09 21:17:33.674565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:33.674397    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kube-rbac-proxy" containerID="cri-o://f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c" gracePeriod=30
Sep 09 21:17:33.837529 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:33.837492    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a657e8e-5745-406d-8385-53cfb56def82" containerID="f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c" exitCode=2
Sep 09 21:17:33.837768 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:33.837570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerDied","Data":"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"}
Sep 09 21:17:34.724274 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:34.724229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 09 21:17:39.212602 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.212574    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:39.314552 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.314476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config\") pod \"1a657e8e-5745-406d-8385-53cfb56def82\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") "
Sep 09 21:17:39.314552 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.314525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert\") pod \"1a657e8e-5745-406d-8385-53cfb56def82\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") "
Sep 09 21:17:39.314552 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.314551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls\") pod \"1a657e8e-5745-406d-8385-53cfb56def82\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") "
Sep 09 21:17:39.314816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.314586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm79l\" (UniqueName: \"kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l\") pod \"1a657e8e-5745-406d-8385-53cfb56def82\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") "
Sep 09 21:17:39.314816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.314649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location\") pod \"1a657e8e-5745-406d-8385-53cfb56def82\" (UID: \"1a657e8e-5745-406d-8385-53cfb56def82\") "
Sep 09 21:17:39.315048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.315004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a657e8e-5745-406d-8385-53cfb56def82" (UID: "1a657e8e-5745-406d-8385-53cfb56def82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:39.315048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.315039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a657e8e-5745-406d-8385-53cfb56def82" (UID: "1a657e8e-5745-406d-8385-53cfb56def82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:39.315757 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.315734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a657e8e-5745-406d-8385-53cfb56def82" (UID: "1a657e8e-5745-406d-8385-53cfb56def82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:39.316823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.316801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a657e8e-5745-406d-8385-53cfb56def82" (UID: "1a657e8e-5745-406d-8385-53cfb56def82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:17:39.316942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.316924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l" (OuterVolumeSpecName: "kube-api-access-dm79l") pod "1a657e8e-5745-406d-8385-53cfb56def82" (UID: "1a657e8e-5745-406d-8385-53cfb56def82"). InnerVolumeSpecName "kube-api-access-dm79l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:17:39.415690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.415650    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a657e8e-5745-406d-8385-53cfb56def82-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.415690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.415681    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.415690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.415691    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a657e8e-5745-406d-8385-53cfb56def82-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.415941 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.415725    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a657e8e-5745-406d-8385-53cfb56def82-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.415941 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.415733    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm79l\" (UniqueName: \"kubernetes.io/projected/1a657e8e-5745-406d-8385-53cfb56def82-kube-api-access-dm79l\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.861314 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.861278    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a657e8e-5745-406d-8385-53cfb56def82" containerID="cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839" exitCode=0
Sep 09 21:17:39.861511 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.861338    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"
Sep 09 21:17:39.861511 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.861361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerDied","Data":"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"}
Sep 09 21:17:39.861511 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.861448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9" event={"ID":"1a657e8e-5745-406d-8385-53cfb56def82","Type":"ContainerDied","Data":"6a0e5c915d4cfc493d9687a3b67ffb80eb1f0230681e488e1ec26c3fdc84717b"}
Sep 09 21:17:39.861511 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.861476    2581 scope.go:117] "RemoveContainer" containerID="f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"
Sep 09 21:17:39.869941 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.869923    2581 scope.go:117] "RemoveContainer" containerID="cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"
Sep 09 21:17:39.877452 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.877429    2581 scope.go:117] "RemoveContainer" containerID="4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"
Sep 09 21:17:39.883572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.883548    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:17:39.885047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.885031    2581 scope.go:117] "RemoveContainer" containerID="f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"
Sep 09 21:17:39.885352 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:17:39.885330    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c\": container with ID starting with f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c not found: ID does not exist" containerID="f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"
Sep 09 21:17:39.885445 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.885364    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c"} err="failed to get container status \"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c\": rpc error: code = NotFound desc = could not find container \"f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c\": container with ID starting with f038c3443f54cd0aed517c030b9e0d4e92bf02bf65f56973af78d24b095b666c not found: ID does not exist"
Sep 09 21:17:39.885445 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.885389    2581 scope.go:117] "RemoveContainer" containerID="cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"
Sep 09 21:17:39.885757 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:17:39.885729    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839\": container with ID starting with cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839 not found: ID does not exist" containerID="cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"
Sep 09 21:17:39.885883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.885764    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839"} err="failed to get container status \"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839\": rpc error: code = NotFound desc = could not find container \"cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839\": container with ID starting with cbbee434f9adf7713fc40a88f97c42d63a36ea6c62261949508a8ca302b0b839 not found: ID does not exist"
Sep 09 21:17:39.885883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.885792    2581 scope.go:117] "RemoveContainer" containerID="4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"
Sep 09 21:17:39.886280 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:17:39.886257    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4\": container with ID starting with 4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4 not found: ID does not exist" containerID="4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"
Sep 09 21:17:39.886380 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.886312    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4"} err="failed to get container status \"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4\": rpc error: code = NotFound desc = could not find container \"4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4\": container with ID starting with 4ff12f527b48ad631a265135920d4ab5d48152f8cc578850a2ea28516603d6d4 not found: ID does not exist"
Sep 09 21:17:39.887479 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:39.887457    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-xdbm9"]
Sep 09 21:17:41.028929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:17:41.028894    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a657e8e-5745-406d-8385-53cfb56def82" path="/var/lib/kubelet/pods/1a657e8e-5745-406d-8385-53cfb56def82/volumes"
Sep 09 21:20:39.750795 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.750712    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751024    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kube-rbac-proxy"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751039    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kube-rbac-proxy"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751052    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="storage-initializer"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="storage-initializer"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751069    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751074    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751128    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kserve-container"
Sep 09 21:20:39.751232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.751142    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a657e8e-5745-406d-8385-53cfb56def82" containerName="kube-rbac-proxy"
Sep 09 21:20:39.754150 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.754132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.758913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.758883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:20:39.759885 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.759858    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:20:39.760064 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.759946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 21:20:39.760064 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.759871    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:20:39.760277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.760244    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:20:39.760559 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.760533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:20:39.762404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.762382    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 21:20:39.776167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.776136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:20:39.846820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.846787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.846997 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.846830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.846997 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.846848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.846997 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.846870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lll62\" (UniqueName: \"kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.846997 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.846939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.947971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.947927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.947971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.947977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.947996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.948018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lll62\" (UniqueName: \"kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.948151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.948384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948744 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.948683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.948744 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.948691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.950522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.950498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:39.958447 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:39.958417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lll62\" (UniqueName: \"kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62\") pod \"isvc-canary-create-predictor-854596548c-bgv5k\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:40.067168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:40.067071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:40.344971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:40.344941    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:20:40.347172 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:20:40.347140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dd4b2b_d349_4347_8c99_01f0e8555252.slice/crio-d1d5732e6e35b485c68e1ae58afcbe9ca02bd1f56f75041ee556c9094cbe554c WatchSource:0}: Error finding container d1d5732e6e35b485c68e1ae58afcbe9ca02bd1f56f75041ee556c9094cbe554c: Status 404 returned error can't find the container with id d1d5732e6e35b485c68e1ae58afcbe9ca02bd1f56f75041ee556c9094cbe554c
Sep 09 21:20:40.500267 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:40.500234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerStarted","Data":"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"}
Sep 09 21:20:40.500459 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:40.500293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerStarted","Data":"d1d5732e6e35b485c68e1ae58afcbe9ca02bd1f56f75041ee556c9094cbe554c"}
Sep 09 21:20:44.516397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:44.516363    2581 generic.go:356] "Generic (PLEG): container finished" podID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerID="7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e" exitCode=0
Sep 09 21:20:44.516779 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:44.516434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerDied","Data":"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"}
Sep 09 21:20:44.517597 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:44.517582    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:20:59.585157 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:59.585122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerStarted","Data":"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"}
Sep 09 21:20:59.585544 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:59.585191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerStarted","Data":"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"}
Sep 09 21:20:59.585544 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:59.585449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:20:59.617875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:20:59.617819    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podStartSLOduration=1.701208631 podStartE2EDuration="20.617801681s" totalImagesPullingTime="14.896558983s" totalInitContainerRuntime="4.020034067s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:59.615362646 +0000 UTC m=+1739.111636699" watchObservedRunningTime="2026-09-09 21:20:59.617801681 +0000 UTC m=+1739.114075776"
Sep 09 21:21:00.589339 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:00.589305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:21:00.590671 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:00.590639    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:01.592768 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:01.592727    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:06.597231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:06.597197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:21:06.597820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:06.597790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:16.598306 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:16.598265    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:26.598247 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:26.598200    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:36.598091 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:36.598053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:38.288935 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.288901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:21:38.292325 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.292309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.296405 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.296383    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 09 21:21:38.296522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.296455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 09 21:21:38.309079 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.309048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.309212 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.309097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd467\" (UniqueName: \"kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.309212 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.309128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.309212 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.309147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.309212 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.309186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.317140 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.317106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:21:38.409566 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.409527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd467\" (UniqueName: \"kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.409566 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.409570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.409807 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.409589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.409807 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.409624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.409807 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.409663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.410141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.410118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.410328 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.410307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.410406 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.410331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.412388 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.412365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.423166 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.423132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd467\" (UniqueName: \"kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.602677 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.602576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:38.897117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:38.897084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:21:38.902159 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:21:38.902130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3415c9_ab2a_4034_9eba_56f5d965522c.slice/crio-bd41df06a00517591b9da0026ccc86963fedf88ec05d9de7db7cf8155b01d424 WatchSource:0}: Error finding container bd41df06a00517591b9da0026ccc86963fedf88ec05d9de7db7cf8155b01d424: Status 404 returned error can't find the container with id bd41df06a00517591b9da0026ccc86963fedf88ec05d9de7db7cf8155b01d424
Sep 09 21:21:39.732437 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:39.732396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerStarted","Data":"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"}
Sep 09 21:21:39.732838 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:39.732455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerStarted","Data":"bd41df06a00517591b9da0026ccc86963fedf88ec05d9de7db7cf8155b01d424"}
Sep 09 21:21:42.745523 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:42.745490    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerID="311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37" exitCode=0
Sep 09 21:21:42.745913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:42.745540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerDied","Data":"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"}
Sep 09 21:21:43.751608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:43.751570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerStarted","Data":"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"}
Sep 09 21:21:43.752005 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:43.751634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerStarted","Data":"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"}
Sep 09 21:21:43.752005 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:43.751850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:43.775864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:43.775814    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podStartSLOduration=2.140783704 podStartE2EDuration="5.775797877s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635014173s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:21:43.775239706 +0000 UTC m=+1783.271513748" watchObservedRunningTime="2026-09-09 21:21:43.775797877 +0000 UTC m=+1783.272071936"
Sep 09 21:21:44.755638 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:44.755607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:44.756927 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:44.756897    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:21:45.758663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:45.758617    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:21:46.598572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:46.598526    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:21:50.763936 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:50.763903    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:21:50.764412 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:50.764384    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:21:56.598394 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:21:56.598304    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:22:00.765203 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:00.765159    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:01.019928 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:01.019846    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:22:01.023218 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:01.023191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:22:03.085254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.085209    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:22:03.085650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.085495    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" containerID="cri-o://4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758" gracePeriod=30
Sep 09 21:22:03.085650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.085532    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" containerID="cri-o://588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471" gracePeriod=30
Sep 09 21:22:03.085814 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.085756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:22:03.834956 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.834920    2581 generic.go:356] "Generic (PLEG): container finished" podID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerID="588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471" exitCode=2
Sep 09 21:22:03.835137 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:03.834978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerDied","Data":"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"}
Sep 09 21:22:06.593977 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:06.593930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:10.765282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:10.765241    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:11.593952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:11.593911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:13.086738 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:13.086671    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 21:22:14.884423 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:14.884390    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:22:14.887828 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:14.887811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:14.890639 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:14.890613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 21:22:14.890772 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:14.890652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 21:22:14.903160 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:14.903130    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:22:15.008115 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.008073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7fc\" (UniqueName: \"kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.008300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.008130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.008300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.008205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.008300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.008240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.008300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.008280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.109590 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.109547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.109813 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.109605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7fc\" (UniqueName: \"kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.109813 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.109779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.110199 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.109926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.110199 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.110007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.110346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.110211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.110449 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.110427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.110651 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.110630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.112369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.112350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.118887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.118860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7fc\" (UniqueName: \"kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-zkbq4\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.197673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.197631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:15.475092 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.474999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:22:15.480195 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:22:15.480165    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1868ccb7_0a8e_4f84_baa9_d09e094597b3.slice/crio-7c85206d70b935ee7492828235250088a6002683cf137c394f9dd30ea9e10f51 WatchSource:0}: Error finding container 7c85206d70b935ee7492828235250088a6002683cf137c394f9dd30ea9e10f51: Status 404 returned error can't find the container with id 7c85206d70b935ee7492828235250088a6002683cf137c394f9dd30ea9e10f51
Sep 09 21:22:15.880948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.880870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerStarted","Data":"d44c54ab858254bdf73fc69aeda033df0ca3a828530dd170abb0334c8d26e49c"}
Sep 09 21:22:15.880948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:15.880925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerStarted","Data":"7c85206d70b935ee7492828235250088a6002683cf137c394f9dd30ea9e10f51"}
Sep 09 21:22:16.593637 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:16.593591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:16.594162 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:16.593782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:22:19.899403 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:19.899372    2581 generic.go:356] "Generic (PLEG): container finished" podID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerID="d44c54ab858254bdf73fc69aeda033df0ca3a828530dd170abb0334c8d26e49c" exitCode=0
Sep 09 21:22:19.899787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:19.899422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerDied","Data":"d44c54ab858254bdf73fc69aeda033df0ca3a828530dd170abb0334c8d26e49c"}
Sep 09 21:22:20.764457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:20.764417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:20.905404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:20.905366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerStarted","Data":"b2ee5d46650348fd9dcde1ad19dcd9728c0133bf1a5a34563d5df65b43fa80a6"}
Sep 09 21:22:20.905904 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:20.905439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerStarted","Data":"c0fa44907adc54cf1897857f213446a67ffc5163390da0e52f23445f6ab7e40b"}
Sep 09 21:22:20.905904 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:20.905661    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:20.931387 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:20.931334    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podStartSLOduration=2.965084828 podStartE2EDuration="6.931315799s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.966230971s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:20.930305522 +0000 UTC m=+1820.426579591" watchObservedRunningTime="2026-09-09 21:22:20.931315799 +0000 UTC m=+1820.427589841"
Sep 09 21:22:21.593385 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:21.593335    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:21.909340 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:21.909306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:21.910731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:21.910689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:22:22.913424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:22.913377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:22:23.087538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:23.087502    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:22:26.593686 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:26.593638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:27.917710 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:27.917667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:22:27.918243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:27.918215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:22:30.765058 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:30.765013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:31.593331 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:31.593281    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 21:22:33.347072 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.347009    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:22:33.473017 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.472966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert\") pod \"18dd4b2b-d349-4347-8c99-01f0e8555252\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") "
Sep 09 21:22:33.473217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lll62\" (UniqueName: \"kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62\") pod \"18dd4b2b-d349-4347-8c99-01f0e8555252\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") "
Sep 09 21:22:33.473217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473076    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location\") pod \"18dd4b2b-d349-4347-8c99-01f0e8555252\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") "
Sep 09 21:22:33.473217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls\") pod \"18dd4b2b-d349-4347-8c99-01f0e8555252\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") "
Sep 09 21:22:33.473217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config\") pod \"18dd4b2b-d349-4347-8c99-01f0e8555252\" (UID: \"18dd4b2b-d349-4347-8c99-01f0e8555252\") "
Sep 09 21:22:33.473414 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18dd4b2b-d349-4347-8c99-01f0e8555252" (UID: "18dd4b2b-d349-4347-8c99-01f0e8555252"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:33.473414 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18dd4b2b-d349-4347-8c99-01f0e8555252" (UID: "18dd4b2b-d349-4347-8c99-01f0e8555252"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:33.473612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.473582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18dd4b2b-d349-4347-8c99-01f0e8555252" (UID: "18dd4b2b-d349-4347-8c99-01f0e8555252"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:33.475112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.475090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18dd4b2b-d349-4347-8c99-01f0e8555252" (UID: "18dd4b2b-d349-4347-8c99-01f0e8555252"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:33.475196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.475179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62" (OuterVolumeSpecName: "kube-api-access-lll62") pod "18dd4b2b-d349-4347-8c99-01f0e8555252" (UID: "18dd4b2b-d349-4347-8c99-01f0e8555252"). InnerVolumeSpecName "kube-api-access-lll62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:33.574545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.574506    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:22:33.574545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.574534    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lll62\" (UniqueName: \"kubernetes.io/projected/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-api-access-lll62\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:22:33.574545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.574544    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18dd4b2b-d349-4347-8c99-01f0e8555252-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:22:33.574545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.574553    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18dd4b2b-d349-4347-8c99-01f0e8555252-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:22:33.574842 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.574561    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18dd4b2b-d349-4347-8c99-01f0e8555252-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:22:33.960346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.960303    2581 generic.go:356] "Generic (PLEG): container finished" podID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerID="4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758" exitCode=137
Sep 09 21:22:33.960522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.960355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"
Sep 09 21:22:33.960522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.960361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerDied","Data":"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"}
Sep 09 21:22:33.960522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.960412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k" event={"ID":"18dd4b2b-d349-4347-8c99-01f0e8555252","Type":"ContainerDied","Data":"d1d5732e6e35b485c68e1ae58afcbe9ca02bd1f56f75041ee556c9094cbe554c"}
Sep 09 21:22:33.960522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.960438    2581 scope.go:117] "RemoveContainer" containerID="588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"
Sep 09 21:22:33.969308 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.969289    2581 scope.go:117] "RemoveContainer" containerID="4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"
Sep 09 21:22:33.976819 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.976795    2581 scope.go:117] "RemoveContainer" containerID="7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"
Sep 09 21:22:33.984024 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984002    2581 scope.go:117] "RemoveContainer" containerID="588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"
Sep 09 21:22:33.984319 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:22:33.984299    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471\": container with ID starting with 588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471 not found: ID does not exist" containerID="588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"
Sep 09 21:22:33.984459 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471"} err="failed to get container status \"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471\": rpc error: code = NotFound desc = could not find container \"588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471\": container with ID starting with 588cb89b0276c20f10364608de30e0bb2e7e4efbeec81cb71fcb6e3792836471 not found: ID does not exist"
Sep 09 21:22:33.984459 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984359    2581 scope.go:117] "RemoveContainer" containerID="4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"
Sep 09 21:22:33.984638 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:22:33.984619    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758\": container with ID starting with 4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758 not found: ID does not exist" containerID="4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"
Sep 09 21:22:33.984685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984644    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758"} err="failed to get container status \"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758\": rpc error: code = NotFound desc = could not find container \"4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758\": container with ID starting with 4514d8ece6ebb3f6a356d35e27872621e64c34e9912930fb4a83cfd96d88a758 not found: ID does not exist"
Sep 09 21:22:33.984685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984659    2581 scope.go:117] "RemoveContainer" containerID="7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"
Sep 09 21:22:33.984971 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:22:33.984952    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e\": container with ID starting with 7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e not found: ID does not exist" containerID="7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"
Sep 09 21:22:33.985034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.984975    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e"} err="failed to get container status \"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e\": rpc error: code = NotFound desc = could not find container \"7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e\": container with ID starting with 7b25af164a7efda1082af9863d2caae8876e1731638d3d228db5564da3c0c72e not found: ID does not exist"
Sep 09 21:22:33.986167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.986149    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:22:33.989419 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:33.989394    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-bgv5k"]
Sep 09 21:22:35.029270 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:35.029235    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" path="/var/lib/kubelet/pods/18dd4b2b-d349-4347-8c99-01f0e8555252/volumes"
Sep 09 21:22:37.918871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:37.918834    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:22:40.765086 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:40.765048    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:47.918601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:47.918563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:22:50.764615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:50.764576    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:22:57.918991 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:22:57.918946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:23:00.764852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:00.764819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:23:07.918561 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:07.918515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:23:08.283502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:08.283402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:23:08.283724 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:08.283658    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" containerID="cri-o://dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763" gracePeriod=30
Sep 09 21:23:08.283837 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:08.283711    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kube-rbac-proxy" containerID="cri-o://af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89" gracePeriod=30
Sep 09 21:23:09.096549 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:09.096516    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerID="af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89" exitCode=2
Sep 09 21:23:09.096931 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:09.096583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerDied","Data":"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"}
Sep 09 21:23:10.759435 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:10.759393    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 09 21:23:10.764792 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:10.764760    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 21:23:13.327341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.327314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:23:13.396333 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config\") pod \"ac3415c9-ab2a-4034-9eba-56f5d965522c\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") "
Sep 09 21:23:13.396502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location\") pod \"ac3415c9-ab2a-4034-9eba-56f5d965522c\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") "
Sep 09 21:23:13.396502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls\") pod \"ac3415c9-ab2a-4034-9eba-56f5d965522c\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") "
Sep 09 21:23:13.396502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert\") pod \"ac3415c9-ab2a-4034-9eba-56f5d965522c\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") "
Sep 09 21:23:13.396502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd467\" (UniqueName: \"kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467\") pod \"ac3415c9-ab2a-4034-9eba-56f5d965522c\" (UID: \"ac3415c9-ab2a-4034-9eba-56f5d965522c\") "
Sep 09 21:23:13.396817 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac3415c9-ab2a-4034-9eba-56f5d965522c" (UID: "ac3415c9-ab2a-4034-9eba-56f5d965522c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:13.396955 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac3415c9-ab2a-4034-9eba-56f5d965522c" (UID: "ac3415c9-ab2a-4034-9eba-56f5d965522c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:13.396955 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.396858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac3415c9-ab2a-4034-9eba-56f5d965522c" (UID: "ac3415c9-ab2a-4034-9eba-56f5d965522c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:13.398653 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.398626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467" (OuterVolumeSpecName: "kube-api-access-cd467") pod "ac3415c9-ab2a-4034-9eba-56f5d965522c" (UID: "ac3415c9-ab2a-4034-9eba-56f5d965522c"). InnerVolumeSpecName "kube-api-access-cd467". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:13.398758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.398690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac3415c9-ab2a-4034-9eba-56f5d965522c" (UID: "ac3415c9-ab2a-4034-9eba-56f5d965522c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:13.497341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.497303    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:13.497341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.497335    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3415c9-ab2a-4034-9eba-56f5d965522c-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:13.497341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.497345    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3415c9-ab2a-4034-9eba-56f5d965522c-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:13.497341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.497354    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3415c9-ab2a-4034-9eba-56f5d965522c-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:13.497599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.497364    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd467\" (UniqueName: \"kubernetes.io/projected/ac3415c9-ab2a-4034-9eba-56f5d965522c-kube-api-access-cd467\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:13.522150 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522116    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:23:13.522394 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522382    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy"
Sep 09 21:23:13.522431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522397    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy"
Sep 09 21:23:13.522431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522417    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container"
Sep 09 21:23:13.522431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522423    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container"
Sep 09 21:23:13.522431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522429    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kube-rbac-proxy"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kube-rbac-proxy"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522442    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="storage-initializer"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522447    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="storage-initializer"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522454    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522459    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522468    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="storage-initializer"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522473    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="storage-initializer"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522516    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kube-rbac-proxy"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522523    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kserve-container"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="18dd4b2b-d349-4347-8c99-01f0e8555252" containerName="kserve-container"
Sep 09 21:23:13.522551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.522534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerName="kube-rbac-proxy"
Sep 09 21:23:13.525982 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.525941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.531615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.531590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 09 21:23:13.531762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.531670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 09 21:23:13.556003 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.555972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:23:13.598668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.598632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7xx\" (UniqueName: \"kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.598860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.598686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.598860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.598733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.598860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.598754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.598860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.598776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.699754 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.699651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.699906 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.699827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7xx\" (UniqueName: \"kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.699906 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.699897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.700015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.699931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.700015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.699952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.700268 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.700247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.700379 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.700360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.700572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.700553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.702503 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.702476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.717156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.717130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7xx\" (UniqueName: \"kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx\") pod \"isvc-tensorflow-predictor-547dfc5669-9qt2c\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:13.835717 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:13.835671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:14.114961 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.114881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:23:14.117254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.117221    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac3415c9-ab2a-4034-9eba-56f5d965522c" containerID="dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763" exitCode=0
Sep 09 21:23:14.117397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.117289    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"
Sep 09 21:23:14.117397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.117304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerDied","Data":"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"}
Sep 09 21:23:14.117397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.117357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn" event={"ID":"ac3415c9-ab2a-4034-9eba-56f5d965522c","Type":"ContainerDied","Data":"bd41df06a00517591b9da0026ccc86963fedf88ec05d9de7db7cf8155b01d424"}
Sep 09 21:23:14.117397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.117385    2581 scope.go:117] "RemoveContainer" containerID="af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"
Sep 09 21:23:14.124779 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:23:14.124749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3411236c_193d_4e3b_a55b_6c08a937f869.slice/crio-de9a34b611f017475e954b5cb23d53cc44529ee2c8e8856c8b2ba50c1ead28a9 WatchSource:0}: Error finding container de9a34b611f017475e954b5cb23d53cc44529ee2c8e8856c8b2ba50c1ead28a9: Status 404 returned error can't find the container with id de9a34b611f017475e954b5cb23d53cc44529ee2c8e8856c8b2ba50c1ead28a9
Sep 09 21:23:14.126613 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.126585    2581 scope.go:117] "RemoveContainer" containerID="dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"
Sep 09 21:23:14.139728 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.139692    2581 scope.go:117] "RemoveContainer" containerID="311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"
Sep 09 21:23:14.151191 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.151174    2581 scope.go:117] "RemoveContainer" containerID="af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"
Sep 09 21:23:14.151504 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:23:14.151485    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89\": container with ID starting with af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89 not found: ID does not exist" containerID="af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"
Sep 09 21:23:14.151541 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.151514    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89"} err="failed to get container status \"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89\": rpc error: code = NotFound desc = could not find container \"af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89\": container with ID starting with af612037f8169e729885990c033b7b2f1d52368890205001d7971118c24f9d89 not found: ID does not exist"
Sep 09 21:23:14.151541 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.151531    2581 scope.go:117] "RemoveContainer" containerID="dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"
Sep 09 21:23:14.151798 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:23:14.151783    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763\": container with ID starting with dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763 not found: ID does not exist" containerID="dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"
Sep 09 21:23:14.151839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.151805    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763"} err="failed to get container status \"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763\": rpc error: code = NotFound desc = could not find container \"dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763\": container with ID starting with dabf0dde568866081699357684f6c9ab33a7cd32a76b6c283e48d276ad446763 not found: ID does not exist"
Sep 09 21:23:14.151839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.151819    2581 scope.go:117] "RemoveContainer" containerID="311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"
Sep 09 21:23:14.152060 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:23:14.152040    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37\": container with ID starting with 311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37 not found: ID does not exist" containerID="311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"
Sep 09 21:23:14.152112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.152065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37"} err="failed to get container status \"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37\": rpc error: code = NotFound desc = could not find container \"311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37\": container with ID starting with 311e66900409db51fc80cf3a8eebee661fddecf29abf85a54b694b75192cdf37 not found: ID does not exist"
Sep 09 21:23:14.158875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.158850    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:23:14.165488 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:14.165464    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-bgwmn"]
Sep 09 21:23:15.031208 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:15.031173    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3415c9-ab2a-4034-9eba-56f5d965522c" path="/var/lib/kubelet/pods/ac3415c9-ab2a-4034-9eba-56f5d965522c/volumes"
Sep 09 21:23:15.126204 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:15.126162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerStarted","Data":"64abafaa364d3987d141ebc05275b24c99eb2b3ce4954099266dfc343038a9ea"}
Sep 09 21:23:15.126376 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:15.126239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerStarted","Data":"de9a34b611f017475e954b5cb23d53cc44529ee2c8e8856c8b2ba50c1ead28a9"}
Sep 09 21:23:17.918563 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:17.918524    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:23:19.142480 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:19.142448    2581 generic.go:356] "Generic (PLEG): container finished" podID="3411236c-193d-4e3b-a55b-6c08a937f869" containerID="64abafaa364d3987d141ebc05275b24c99eb2b3ce4954099266dfc343038a9ea" exitCode=0
Sep 09 21:23:19.142845 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:19.142522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerDied","Data":"64abafaa364d3987d141ebc05275b24c99eb2b3ce4954099266dfc343038a9ea"}
Sep 09 21:23:23.162419 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:23.162385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerStarted","Data":"d4bcb3d1b2f23f9fa5cbd88e829dc0eabb12ce9840024126c68d02a54c7e6480"}
Sep 09 21:23:23.162825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:23.162463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerStarted","Data":"935faf5bc3ab73247d076dce000b2a54cfbabdc059dbcf0800013c10092d7210"}
Sep 09 21:23:23.162825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:23.162667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:23.185042 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:23.184990    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podStartSLOduration=1.692618685 podStartE2EDuration="10.184972504s" totalImagesPullingTime="3.871806407s" totalInitContainerRuntime="4.620547412s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:23.182866988 +0000 UTC m=+1882.679141051" watchObservedRunningTime="2026-09-09 21:23:23.184972504 +0000 UTC m=+1882.681246546"
Sep 09 21:23:24.170538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:24.170492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:24.170959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:24.170632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 21:23:25.172359 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:25.172309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 21:23:27.918977 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:27.918938    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:23:30.177583 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:30.177553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:30.178088 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:30.178060    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 09 21:23:32.024968 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:32.024921    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 21:23:40.179157 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:40.179126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:23:42.025880 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:42.025840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:23:44.192552 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.192514    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:23:44.193046 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.192814    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container" containerID="cri-o://c0fa44907adc54cf1897857f213446a67ffc5163390da0e52f23445f6ab7e40b" gracePeriod=30
Sep 09 21:23:44.193046 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.192878    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kube-rbac-proxy" containerID="cri-o://b2ee5d46650348fd9dcde1ad19dcd9728c0133bf1a5a34563d5df65b43fa80a6" gracePeriod=30
Sep 09 21:23:44.878645 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.878607    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:23:44.881991 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.881975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:44.884749 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.884728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 21:23:44.897747 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.897710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:23:44.950252 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.950209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:44.950439 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.950262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:44.950439 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.950320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:44.950439 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.950387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:44.950610 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:44.950495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvn2\" (UniqueName: \"kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvn2\" (UniqueName: \"kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.051745 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.051721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.052023 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.052001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.052099 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.052009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.053749 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.053731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.060568 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.060545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvn2\" (UniqueName: \"kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-mtbbg\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.191648 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.191609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:45.250052 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.250024    2581 generic.go:356] "Generic (PLEG): container finished" podID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerID="b2ee5d46650348fd9dcde1ad19dcd9728c0133bf1a5a34563d5df65b43fa80a6" exitCode=2
Sep 09 21:23:45.250448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.250092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerDied","Data":"b2ee5d46650348fd9dcde1ad19dcd9728c0133bf1a5a34563d5df65b43fa80a6"}
Sep 09 21:23:45.471356 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:45.471271    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:23:45.474452 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:23:45.474418    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49af8a63_7e2e_4e42_a6c2_87b0817cdcf1.slice/crio-814932ba24a7a36629edbec8005e5c70a187ddcddfa73e80ff59b27e3cea4ec9 WatchSource:0}: Error finding container 814932ba24a7a36629edbec8005e5c70a187ddcddfa73e80ff59b27e3cea4ec9: Status 404 returned error can't find the container with id 814932ba24a7a36629edbec8005e5c70a187ddcddfa73e80ff59b27e3cea4ec9
Sep 09 21:23:46.256030 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:46.255992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerStarted","Data":"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"}
Sep 09 21:23:46.256412 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:46.256076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerStarted","Data":"814932ba24a7a36629edbec8005e5c70a187ddcddfa73e80ff59b27e3cea4ec9"}
Sep 09 21:23:47.914000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:47.913961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 09 21:23:49.271807 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.271775    2581 generic.go:356] "Generic (PLEG): container finished" podID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerID="c0fa44907adc54cf1897857f213446a67ffc5163390da0e52f23445f6ab7e40b" exitCode=0
Sep 09 21:23:49.272188 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.271845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerDied","Data":"c0fa44907adc54cf1897857f213446a67ffc5163390da0e52f23445f6ab7e40b"}
Sep 09 21:23:49.273668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.273627    2581 generic.go:356] "Generic (PLEG): container finished" podID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerID="8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a" exitCode=0
Sep 09 21:23:49.273858 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.273675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerDied","Data":"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"}
Sep 09 21:23:49.343480 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.343452    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:23:49.387223 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph7fc\" (UniqueName: \"kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc\") pod \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") "
Sep 09 21:23:49.387361 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config\") pod \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") "
Sep 09 21:23:49.387361 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls\") pod \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") "
Sep 09 21:23:49.387361 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location\") pod \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") "
Sep 09 21:23:49.387549 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert\") pod \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\" (UID: \"1868ccb7-0a8e-4f84-baa9-d09e094597b3\") "
Sep 09 21:23:49.387653 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1868ccb7-0a8e-4f84-baa9-d09e094597b3" (UID: "1868ccb7-0a8e-4f84-baa9-d09e094597b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:49.387837 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1868ccb7-0a8e-4f84-baa9-d09e094597b3" (UID: "1868ccb7-0a8e-4f84-baa9-d09e094597b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:49.387946 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.387920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1868ccb7-0a8e-4f84-baa9-d09e094597b3" (UID: "1868ccb7-0a8e-4f84-baa9-d09e094597b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:49.389477 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.389455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1868ccb7-0a8e-4f84-baa9-d09e094597b3" (UID: "1868ccb7-0a8e-4f84-baa9-d09e094597b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:49.389565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.389484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc" (OuterVolumeSpecName: "kube-api-access-ph7fc") pod "1868ccb7-0a8e-4f84-baa9-d09e094597b3" (UID: "1868ccb7-0a8e-4f84-baa9-d09e094597b3"). InnerVolumeSpecName "kube-api-access-ph7fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:49.488502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.488467    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1868ccb7-0a8e-4f84-baa9-d09e094597b3-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.488502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.488499    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.488502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.488509    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.488502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.488518    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7fc\" (UniqueName: \"kubernetes.io/projected/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-api-access-ph7fc\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.488803 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:49.488527    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1868ccb7-0a8e-4f84-baa9-d09e094597b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:23:50.279538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.279499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerStarted","Data":"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"}
Sep 09 21:23:50.279947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.279580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerStarted","Data":"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"}
Sep 09 21:23:50.279947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.279905    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:50.280066 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.280041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:50.281541 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.281519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4" event={"ID":"1868ccb7-0a8e-4f84-baa9-d09e094597b3","Type":"ContainerDied","Data":"7c85206d70b935ee7492828235250088a6002683cf137c394f9dd30ea9e10f51"}
Sep 09 21:23:50.281675 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.281561    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"
Sep 09 21:23:50.281675 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.281570    2581 scope.go:117] "RemoveContainer" containerID="b2ee5d46650348fd9dcde1ad19dcd9728c0133bf1a5a34563d5df65b43fa80a6"
Sep 09 21:23:50.281675 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.281570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:23:50.294948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.290230    2581 scope.go:117] "RemoveContainer" containerID="c0fa44907adc54cf1897857f213446a67ffc5163390da0e52f23445f6ab7e40b"
Sep 09 21:23:50.303305 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.303279    2581 scope.go:117] "RemoveContainer" containerID="d44c54ab858254bdf73fc69aeda033df0ca3a828530dd170abb0334c8d26e49c"
Sep 09 21:23:50.303601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.303557    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podStartSLOduration=2.656272703 podStartE2EDuration="6.30354394s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.647271237s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:50.301642384 +0000 UTC m=+1909.797916426" watchObservedRunningTime="2026-09-09 21:23:50.30354394 +0000 UTC m=+1909.799817982"
Sep 09 21:23:50.318367 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.318339    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:23:50.325375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:50.325346    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-zkbq4"]
Sep 09 21:23:51.029934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:51.029902    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" path="/var/lib/kubelet/pods/1868ccb7-0a8e-4f84-baa9-d09e094597b3/volumes"
Sep 09 21:23:51.287099 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:51.287014    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:23:55.098850 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.098817    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:23:55.099294 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.099065    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container" containerID="cri-o://935faf5bc3ab73247d076dce000b2a54cfbabdc059dbcf0800013c10092d7210" gracePeriod=30
Sep 09 21:23:55.099294 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.099118    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" containerID="cri-o://d4bcb3d1b2f23f9fa5cbd88e829dc0eabb12ce9840024126c68d02a54c7e6480" gracePeriod=30
Sep 09 21:23:55.172791 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.172752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:23:55.302942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.302906    2581 generic.go:356] "Generic (PLEG): container finished" podID="3411236c-193d-4e3b-a55b-6c08a937f869" containerID="d4bcb3d1b2f23f9fa5cbd88e829dc0eabb12ce9840024126c68d02a54c7e6480" exitCode=2
Sep 09 21:23:55.303107 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:55.302981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerDied","Data":"d4bcb3d1b2f23f9fa5cbd88e829dc0eabb12ce9840024126c68d02a54c7e6480"}
Sep 09 21:23:56.291883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:56.291848    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:23:56.292368 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:23:56.292339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:00.172716 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.172650    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:00.319735 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.319684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:24:00.320060 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320044    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="storage-initializer"
Sep 09 21:24:00.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320063    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="storage-initializer"
Sep 09 21:24:00.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320073    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container"
Sep 09 21:24:00.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320079    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container"
Sep 09 21:24:00.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kube-rbac-proxy"
Sep 09 21:24:00.320123 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320095    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kube-rbac-proxy"
Sep 09 21:24:00.320273 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320148    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kserve-container"
Sep 09 21:24:00.320273 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.320155    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1868ccb7-0a8e-4f84-baa9-d09e094597b3" containerName="kube-rbac-proxy"
Sep 09 21:24:00.323291 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.323270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.327015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.326991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 09 21:24:00.327142 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.327038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:24:00.339023 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.338988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:24:00.377357 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.377321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67sc6\" (UniqueName: \"kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.377560 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.377369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.377560 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.377425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.377560 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.377480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.377560 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.377543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.478900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.478805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.478900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.478863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.478900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.478901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67sc6\" (UniqueName: \"kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.479190 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.478925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.479190 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.478944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.479390 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.479355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.479557 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.479535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.479663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.479600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.481453 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.481433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.497300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.497271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67sc6\" (UniqueName: \"kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.634307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.634265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:00.918831 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:00.918799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:24:00.922597 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:24:00.922571    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bc99c0_e795_4b67_82d7_aefb99b36262.slice/crio-58712663c7a8aa926f9e474ed595f39be203067e163d01969fd7fa937398757c WatchSource:0}: Error finding container 58712663c7a8aa926f9e474ed595f39be203067e163d01969fd7fa937398757c: Status 404 returned error can't find the container with id 58712663c7a8aa926f9e474ed595f39be203067e163d01969fd7fa937398757c
Sep 09 21:24:01.332169 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:01.332069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerStarted","Data":"a12252a2fb56a4391f8efe1ad85e7c8409071988085886bf1dc6b5eac0182d3b"}
Sep 09 21:24:01.332169 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:01.332139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerStarted","Data":"58712663c7a8aa926f9e474ed595f39be203067e163d01969fd7fa937398757c"}
Sep 09 21:24:05.173432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:05.173378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:05.173874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:05.173544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:24:06.292871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:06.292830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:06.351442 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:06.351406    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerID="a12252a2fb56a4391f8efe1ad85e7c8409071988085886bf1dc6b5eac0182d3b" exitCode=0
Sep 09 21:24:06.351615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:06.351457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerDied","Data":"a12252a2fb56a4391f8efe1ad85e7c8409071988085886bf1dc6b5eac0182d3b"}
Sep 09 21:24:07.357073 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.357039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerStarted","Data":"a4d04d484e6bd09d9acdde66c9b39fe27d39dca6d274fbe2c36b02bfdecc27e9"}
Sep 09 21:24:07.357466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.357100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerStarted","Data":"5a2308a163911ddbf7686cce0f8ca192e5706d73664565fb25543bc5e612d79b"}
Sep 09 21:24:07.357466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.357372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:07.357547 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.357524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:07.358750 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.358726    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 21:24:07.386887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:07.386838    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podStartSLOduration=2.867002419 podStartE2EDuration="7.386823472s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.519821053s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:07.384633074 +0000 UTC m=+1926.880907127" watchObservedRunningTime="2026-09-09 21:24:07.386823472 +0000 UTC m=+1926.883097514"
Sep 09 21:24:08.360731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:08.360680    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 21:24:10.172585 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:10.172539    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:13.365286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:13.365254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:13.365920 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:13.365896    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 21:24:15.172623 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:15.172573    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:16.292355 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:16.292318    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:20.172589 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:20.172541    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:23.366866 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:23.366837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:25.173418 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.173369    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 09 21:24:25.426898 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.426862    2581 generic.go:356] "Generic (PLEG): container finished" podID="3411236c-193d-4e3b-a55b-6c08a937f869" containerID="935faf5bc3ab73247d076dce000b2a54cfbabdc059dbcf0800013c10092d7210" exitCode=137
Sep 09 21:24:25.427142 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.426929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerDied","Data":"935faf5bc3ab73247d076dce000b2a54cfbabdc059dbcf0800013c10092d7210"}
Sep 09 21:24:25.733792 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.733766    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:24:25.895944 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.895903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location\") pod \"3411236c-193d-4e3b-a55b-6c08a937f869\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") "
Sep 09 21:24:25.896129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.895958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls\") pod \"3411236c-193d-4e3b-a55b-6c08a937f869\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") "
Sep 09 21:24:25.896129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.895982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config\") pod \"3411236c-193d-4e3b-a55b-6c08a937f869\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") "
Sep 09 21:24:25.896129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.895998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert\") pod \"3411236c-193d-4e3b-a55b-6c08a937f869\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") "
Sep 09 21:24:25.896129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.896039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7xx\" (UniqueName: \"kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx\") pod \"3411236c-193d-4e3b-a55b-6c08a937f869\" (UID: \"3411236c-193d-4e3b-a55b-6c08a937f869\") "
Sep 09 21:24:25.896387 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.896360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3411236c-193d-4e3b-a55b-6c08a937f869" (UID: "3411236c-193d-4e3b-a55b-6c08a937f869"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:24:25.896471 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.896447    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3411236c-193d-4e3b-a55b-6c08a937f869" (UID: "3411236c-193d-4e3b-a55b-6c08a937f869"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:24:25.898224 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.898191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3411236c-193d-4e3b-a55b-6c08a937f869" (UID: "3411236c-193d-4e3b-a55b-6c08a937f869"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:25.898340 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.898253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx" (OuterVolumeSpecName: "kube-api-access-nw7xx") pod "3411236c-193d-4e3b-a55b-6c08a937f869" (UID: "3411236c-193d-4e3b-a55b-6c08a937f869"). InnerVolumeSpecName "kube-api-access-nw7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:25.906527 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.906501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3411236c-193d-4e3b-a55b-6c08a937f869" (UID: "3411236c-193d-4e3b-a55b-6c08a937f869"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:25.997484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.997395    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3411236c-193d-4e3b-a55b-6c08a937f869-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:24:25.997484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.997429    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3411236c-193d-4e3b-a55b-6c08a937f869-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:24:25.997484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.997439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:24:25.997484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.997448    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3411236c-193d-4e3b-a55b-6c08a937f869-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:24:25.997484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:25.997456    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7xx\" (UniqueName: \"kubernetes.io/projected/3411236c-193d-4e3b-a55b-6c08a937f869-kube-api-access-nw7xx\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:24:26.292827 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.292739    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:26.432749 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.432718    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"
Sep 09 21:24:26.432749 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.432731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c" event={"ID":"3411236c-193d-4e3b-a55b-6c08a937f869","Type":"ContainerDied","Data":"de9a34b611f017475e954b5cb23d53cc44529ee2c8e8856c8b2ba50c1ead28a9"}
Sep 09 21:24:26.432939 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.432798    2581 scope.go:117] "RemoveContainer" containerID="d4bcb3d1b2f23f9fa5cbd88e829dc0eabb12ce9840024126c68d02a54c7e6480"
Sep 09 21:24:26.441320 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.441301    2581 scope.go:117] "RemoveContainer" containerID="935faf5bc3ab73247d076dce000b2a54cfbabdc059dbcf0800013c10092d7210"
Sep 09 21:24:26.448519 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.448501    2581 scope.go:117] "RemoveContainer" containerID="64abafaa364d3987d141ebc05275b24c99eb2b3ce4954099266dfc343038a9ea"
Sep 09 21:24:26.456616 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.456595    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:24:26.461942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:26.461919    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9qt2c"]
Sep 09 21:24:27.030395 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:27.030368    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" path="/var/lib/kubelet/pods/3411236c-193d-4e3b-a55b-6c08a937f869/volumes"
Sep 09 21:24:36.292719 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:36.292662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:42.111308 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:42.111268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:24:42.111810 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:42.111528    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container" containerID="cri-o://5a2308a163911ddbf7686cce0f8ca192e5706d73664565fb25543bc5e612d79b" gracePeriod=30
Sep 09 21:24:42.111810 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:42.111568    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" containerID="cri-o://a4d04d484e6bd09d9acdde66c9b39fe27d39dca6d274fbe2c36b02bfdecc27e9" gracePeriod=30
Sep 09 21:24:42.496580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:42.496538    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerID="a4d04d484e6bd09d9acdde66c9b39fe27d39dca6d274fbe2c36b02bfdecc27e9" exitCode=2
Sep 09 21:24:42.496782 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:42.496604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerDied","Data":"a4d04d484e6bd09d9acdde66c9b39fe27d39dca6d274fbe2c36b02bfdecc27e9"}
Sep 09 21:24:43.361628 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:43.361535    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:24:46.292795 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:46.292749    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:47.466818 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.466784    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467070    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="storage-initializer"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467081    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="storage-initializer"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467122    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467170    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kube-rbac-proxy"
Sep 09 21:24:47.467225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.467177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3411236c-193d-4e3b-a55b-6c08a937f869" containerName="kserve-container"
Sep 09 21:24:47.470291 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.470264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.477398 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.477370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 09 21:24:47.477787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.477771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 09 21:24:47.507864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.507824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:24:47.571315 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.571268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.571508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.571322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.571508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.571401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7ln\" (UniqueName: \"kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.571508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.571463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.571508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.571504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.672310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.672272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.672507 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.672330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.672507 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:24:47.672440    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 09 21:24:47.672507 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:24:47.672506    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls podName:925de38a-3d5d-4dfa-98bf-7dcfd090c934 nodeName:}" failed. No retries permitted until 2026-09-09 21:24:48.172488364 +0000 UTC m=+1967.668762384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls") pod "isvc-triton-predictor-5d7db58676-xn62s" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934") : secret "isvc-triton-predictor-serving-cert" not found
Sep 09 21:24:47.672672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.672446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.672672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.672582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7ln\" (UniqueName: \"kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.672672 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.672638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.673068 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.673045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.673156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.673070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.673156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.673050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:47.683570 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:47.683536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7ln\" (UniqueName: \"kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:48.178071 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:48.178019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:48.180445 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:48.180424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xn62s\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:48.361912 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:48.361864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:24:48.379619 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:48.379585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:24:48.664813 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:48.664782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:24:48.668581 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:24:48.668550    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925de38a_3d5d_4dfa_98bf_7dcfd090c934.slice/crio-1bc5e14effc3a2adef2967e16cf6f793c7bab1dff35c4d7e2e850666caad02f9 WatchSource:0}: Error finding container 1bc5e14effc3a2adef2967e16cf6f793c7bab1dff35c4d7e2e850666caad02f9: Status 404 returned error can't find the container with id 1bc5e14effc3a2adef2967e16cf6f793c7bab1dff35c4d7e2e850666caad02f9
Sep 09 21:24:49.524046 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:49.524008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerStarted","Data":"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"}
Sep 09 21:24:49.524254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:49.524068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerStarted","Data":"1bc5e14effc3a2adef2967e16cf6f793c7bab1dff35c4d7e2e850666caad02f9"}
Sep 09 21:24:53.361978 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:53.361932    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:24:53.362444 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:53.362065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:24:53.541773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:53.541739    2581 generic.go:356] "Generic (PLEG): container finished" podID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerID="c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208" exitCode=0
Sep 09 21:24:53.541947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:53.541814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerDied","Data":"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"}
Sep 09 21:24:56.293496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:56.292920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:24:58.361899 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:24:58.361849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:25:03.362108 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:03.362055    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:25:06.293406 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:06.293342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:25:08.362264 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:08.361851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 21:25:12.676892 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.676829    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerID="5a2308a163911ddbf7686cce0f8ca192e5706d73664565fb25543bc5e612d79b" exitCode=137
Sep 09 21:25:12.677344 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.676930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerDied","Data":"5a2308a163911ddbf7686cce0f8ca192e5706d73664565fb25543bc5e612d79b"}
Sep 09 21:25:12.829290 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.829208    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:25:12.904014 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.903955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert\") pod \"93bc99c0-e795-4b67-82d7-aefb99b36262\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") "
Sep 09 21:25:12.904213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config\") pod \"93bc99c0-e795-4b67-82d7-aefb99b36262\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") "
Sep 09 21:25:12.904213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67sc6\" (UniqueName: \"kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6\") pod \"93bc99c0-e795-4b67-82d7-aefb99b36262\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") "
Sep 09 21:25:12.904213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location\") pod \"93bc99c0-e795-4b67-82d7-aefb99b36262\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") "
Sep 09 21:25:12.904213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls\") pod \"93bc99c0-e795-4b67-82d7-aefb99b36262\" (UID: \"93bc99c0-e795-4b67-82d7-aefb99b36262\") "
Sep 09 21:25:12.904447 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93bc99c0-e795-4b67-82d7-aefb99b36262" (UID: "93bc99c0-e795-4b67-82d7-aefb99b36262"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:12.904781 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.904684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93bc99c0-e795-4b67-82d7-aefb99b36262" (UID: "93bc99c0-e795-4b67-82d7-aefb99b36262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:12.906879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.906840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93bc99c0-e795-4b67-82d7-aefb99b36262" (UID: "93bc99c0-e795-4b67-82d7-aefb99b36262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:12.907494 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.907468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6" (OuterVolumeSpecName: "kube-api-access-67sc6") pod "93bc99c0-e795-4b67-82d7-aefb99b36262" (UID: "93bc99c0-e795-4b67-82d7-aefb99b36262"). InnerVolumeSpecName "kube-api-access-67sc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:12.916535 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:12.916500    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93bc99c0-e795-4b67-82d7-aefb99b36262" (UID: "93bc99c0-e795-4b67-82d7-aefb99b36262"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:13.005275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.005162    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:13.005275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.005199    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67sc6\" (UniqueName: \"kubernetes.io/projected/93bc99c0-e795-4b67-82d7-aefb99b36262-kube-api-access-67sc6\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:13.005275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.005214    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bc99c0-e795-4b67-82d7-aefb99b36262-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:13.005275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.005230    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93bc99c0-e795-4b67-82d7-aefb99b36262-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:13.005275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.005250    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bc99c0-e795-4b67-82d7-aefb99b36262-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:13.684884 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.684833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz" event={"ID":"93bc99c0-e795-4b67-82d7-aefb99b36262","Type":"ContainerDied","Data":"58712663c7a8aa926f9e474ed595f39be203067e163d01969fd7fa937398757c"}
Sep 09 21:25:13.685575 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.685437    2581 scope.go:117] "RemoveContainer" containerID="a4d04d484e6bd09d9acdde66c9b39fe27d39dca6d274fbe2c36b02bfdecc27e9"
Sep 09 21:25:13.685575 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.684859    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"
Sep 09 21:25:13.697434 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.697416    2581 scope.go:117] "RemoveContainer" containerID="5a2308a163911ddbf7686cce0f8ca192e5706d73664565fb25543bc5e612d79b"
Sep 09 21:25:13.707663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.707631    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:25:13.708416 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.708400    2581 scope.go:117] "RemoveContainer" containerID="a12252a2fb56a4391f8efe1ad85e7c8409071988085886bf1dc6b5eac0182d3b"
Sep 09 21:25:13.713018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:13.712991    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-m5ztz"]
Sep 09 21:25:15.033279 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:15.033245    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" path="/var/lib/kubelet/pods/93bc99c0-e795-4b67-82d7-aefb99b36262/volumes"
Sep 09 21:25:16.292898 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:16.292869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:25:25.010596 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:25.010557    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:25:25.011182 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:25.010880    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" containerID="cri-o://074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e" gracePeriod=30
Sep 09 21:25:25.011182 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:25.010965    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy" containerID="cri-o://7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075" gracePeriod=30
Sep 09 21:25:25.751319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:25.751204    2581 generic.go:356] "Generic (PLEG): container finished" podID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerID="7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075" exitCode=2
Sep 09 21:25:25.751319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:25.751245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerDied","Data":"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"}
Sep 09 21:25:26.287813 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:26.287747    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 09 21:25:26.293116 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:26.293068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 09 21:25:30.231262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231220    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231649    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy"
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231665    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy"
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231691    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="storage-initializer"
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231714    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="storage-initializer"
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container"
Sep 09 21:25:30.231806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container"
Sep 09 21:25:30.232138 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231811    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kserve-container"
Sep 09 21:25:30.232138 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.231823    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bc99c0-e795-4b67-82d7-aefb99b36262" containerName="kube-rbac-proxy"
Sep 09 21:25:30.257506 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.257466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:25:30.257952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.257573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.260326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.260296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 21:25:30.260769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.260571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 21:25:30.314401 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.312992    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:25:30.335183 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.334269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:25:30.335183 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.334433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.340738 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.340504    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 21:25:30.359909 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.359873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.360107 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.359930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.360107 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.360013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl7z\" (UniqueName: \"kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.360107 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.360081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.360277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.360130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.461285 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:30.461438    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:30.461516    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls podName:b6ba385d-3ddd-4d3c-ae27-ffb64dad0625 nodeName:}" failed. No retries permitted until 2026-09-09 21:25:30.961494268 +0000 UTC m=+2010.457768289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 21:25:30.461600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.462113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7bw\" (UniqueName: \"kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.462113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.461659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl7z\" (UniqueName: \"kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.462948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.462581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.462948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.462940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.463174 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.463138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.472286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.472241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl7z\" (UniqueName: \"kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.562634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.562742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.562795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.562843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:30.562856    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.562869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7bw\" (UniqueName: \"kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:30.562941    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls podName:4ebf755b-c2ce-4bbd-a61e-674a0aff9667 nodeName:}" failed. No retries permitted until 2026-09-09 21:25:31.062921407 +0000 UTC m=+2010.559195443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls") pod "isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:25:30.563365 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.563204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.564010 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.563817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.564535 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.564489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.575160 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.575104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7bw\" (UniqueName: \"kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:30.967854 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.967788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:30.971173 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:30.971117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:31.068841 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.068790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:31.071925 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.071866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-nsgt8\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:31.173047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.172979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:25:31.251501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.251253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:25:31.287356 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.287313    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 09 21:25:31.448650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.448568    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:25:31.493209 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.493178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:25:31.496760 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:25:31.496721    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ba385d_3ddd_4d3c_ae27_ffb64dad0625.slice/crio-8e8a95b7d80df0aed9d8e9865ec09a344bc5d27d92c8fae0014911e72c629976 WatchSource:0}: Error finding container 8e8a95b7d80df0aed9d8e9865ec09a344bc5d27d92c8fae0014911e72c629976: Status 404 returned error can't find the container with id 8e8a95b7d80df0aed9d8e9865ec09a344bc5d27d92c8fae0014911e72c629976
Sep 09 21:25:31.575284 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location\") pod \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") "
Sep 09 21:25:31.575479 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnvn2\" (UniqueName: \"kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2\") pod \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") "
Sep 09 21:25:31.575479 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls\") pod \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") "
Sep 09 21:25:31.575479 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config\") pod \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") "
Sep 09 21:25:31.575479 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575465    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert\") pod \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\" (UID: \"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1\") "
Sep 09 21:25:31.575733 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.575681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" (UID: "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:31.576407 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.576379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" (UID: "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:31.576558 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.576526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" (UID: "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:31.579397 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.579369    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2" (OuterVolumeSpecName: "kube-api-access-pnvn2") pod "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" (UID: "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1"). InnerVolumeSpecName "kube-api-access-pnvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:31.582028 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.581851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" (UID: "49af8a63-7e2e-4e42-a6c2-87b0817cdcf1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:31.596835 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.596790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:25:31.677158 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.677121    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvn2\" (UniqueName: \"kubernetes.io/projected/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-api-access-pnvn2\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:31.677158 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.677160    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:31.677374 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.677175    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:31.677374 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.677189    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:31.677374 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.677204    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:25:31.791277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.786140    2581 generic.go:356] "Generic (PLEG): container finished" podID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerID="074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e" exitCode=0
Sep 09 21:25:31.791277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.786226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerDied","Data":"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"}
Sep 09 21:25:31.791277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.786289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg" event={"ID":"49af8a63-7e2e-4e42-a6c2-87b0817cdcf1","Type":"ContainerDied","Data":"814932ba24a7a36629edbec8005e5c70a187ddcddfa73e80ff59b27e3cea4ec9"}
Sep 09 21:25:31.791277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.786326    2581 scope.go:117] "RemoveContainer" containerID="7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"
Sep 09 21:25:31.791277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.786521    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"
Sep 09 21:25:31.797504 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.796725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerStarted","Data":"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"}
Sep 09 21:25:31.797504 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.796800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerStarted","Data":"8e8a95b7d80df0aed9d8e9865ec09a344bc5d27d92c8fae0014911e72c629976"}
Sep 09 21:25:31.800577 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.800533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerStarted","Data":"0578a64c17dda60a3d9d89c944d15a0d11af9fc1a8860c50596785b65cfecf1d"}
Sep 09 21:25:31.800750 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.800617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerStarted","Data":"a0590383d8d1fca96fd4ce1bd7183394c7f07db52a13090c5422059913f736ac"}
Sep 09 21:25:31.808582 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.808554    2581 scope.go:117] "RemoveContainer" containerID="074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"
Sep 09 21:25:31.822673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.822648    2581 scope.go:117] "RemoveContainer" containerID="8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"
Sep 09 21:25:31.835791 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.835761    2581 scope.go:117] "RemoveContainer" containerID="7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"
Sep 09 21:25:31.837423 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:31.836947    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075\": container with ID starting with 7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075 not found: ID does not exist" containerID="7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"
Sep 09 21:25:31.837423 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.837015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075"} err="failed to get container status \"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075\": rpc error: code = NotFound desc = could not find container \"7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075\": container with ID starting with 7a705c17720da5e5dd5810ab965d078629070941c7bb8341b13214e54cbbb075 not found: ID does not exist"
Sep 09 21:25:31.837423 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.837040    2581 scope.go:117] "RemoveContainer" containerID="074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"
Sep 09 21:25:31.837685 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:31.837425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e\": container with ID starting with 074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e not found: ID does not exist" containerID="074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"
Sep 09 21:25:31.837685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.837461    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e"} err="failed to get container status \"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e\": rpc error: code = NotFound desc = could not find container \"074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e\": container with ID starting with 074fe54257a72fd554580dfc27fc0e0fce34329006e7ffbb155f37f31f93880e not found: ID does not exist"
Sep 09 21:25:31.837685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.837483    2581 scope.go:117] "RemoveContainer" containerID="8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"
Sep 09 21:25:31.838126 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:25:31.838063    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a\": container with ID starting with 8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a not found: ID does not exist" containerID="8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"
Sep 09 21:25:31.838126 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.838097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a"} err="failed to get container status \"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a\": rpc error: code = NotFound desc = could not find container \"8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a\": container with ID starting with 8ea8459c74b17adceca1aefedc337f7c841034bc6ab8afdcab573f821bd47c1a not found: ID does not exist"
Sep 09 21:25:31.839027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.839000    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:25:31.845189 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:31.845119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-mtbbg"]
Sep 09 21:25:33.032396 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:33.032228    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" path="/var/lib/kubelet/pods/49af8a63-7e2e-4e42-a6c2-87b0817cdcf1/volumes"
Sep 09 21:25:36.829668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:36.829626    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerID="20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8" exitCode=0
Sep 09 21:25:36.830165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:36.829724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerDied","Data":"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"}
Sep 09 21:25:36.834322 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:36.832934    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerID="0578a64c17dda60a3d9d89c944d15a0d11af9fc1a8860c50596785b65cfecf1d" exitCode=0
Sep 09 21:25:36.834322 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:25:36.833025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerDied","Data":"0578a64c17dda60a3d9d89c944d15a0d11af9fc1a8860c50596785b65cfecf1d"}
Sep 09 21:26:09.987615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:09.987574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerStarted","Data":"d10ac35323361f96d8587eec3467216b266ec40688084ed189b3dffeed7a6ec0"}
Sep 09 21:26:14.006375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:14.006330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerStarted","Data":"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"}
Sep 09 21:26:20.035988 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:20.035950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerStarted","Data":"4d5dc34774eb465ff4118a2d015720ec08d24872ef3a6d613d90417bc95fdfd9"}
Sep 09 21:26:20.036512 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:20.036095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:26:20.058940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:20.058869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podStartSLOduration=45.822174271 podStartE2EDuration="50.058848473s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.236674202s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:20.057470674 +0000 UTC m=+2059.553744763" watchObservedRunningTime="2026-09-09 21:26:20.058848473 +0000 UTC m=+2059.555122519"
Sep 09 21:26:21.039969 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:21.039937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:26:21.041605 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:21.041572    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:22.043934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:22.043893    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:27.049278 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:27.049187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:26:27.049972 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:27.049936    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:37.050817 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:37.050770    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:47.050630 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:47.050539    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:47.150116 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:47.150080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerStarted","Data":"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"}
Sep 09 21:26:47.150290 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:47.150243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:26:47.174806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:47.174754    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podStartSLOduration=72.83983034 podStartE2EDuration="1m17.174735834s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.334905494s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:47.172133856 +0000 UTC m=+2086.668407899" watchObservedRunningTime="2026-09-09 21:26:47.174735834 +0000 UTC m=+2086.671009934"
Sep 09 21:26:48.156524 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.156482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerStarted","Data":"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"}
Sep 09 21:26:48.156976 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.156582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:26:48.156976 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.156942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:26:48.157054 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.156990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:26:48.157054 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.157010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerStarted","Data":"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"}
Sep 09 21:26:48.157945 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.157917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 21:26:48.158076 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.157992    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:26:48.178434 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:48.178390    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podStartSLOduration=2.846154854 podStartE2EDuration="2m1.178374554s" totalImagesPullingTime="1m54.343541716s" totalInitContainerRuntime="3.988677984s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:48.176926494 +0000 UTC m=+2087.673200555" watchObservedRunningTime="2026-09-09 21:26:48.178374554 +0000 UTC m=+2087.674648595"
Sep 09 21:26:49.160779 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:49.160731    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:26:49.161310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:49.160835    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 21:26:50.164740 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:50.164679    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 21:26:54.164971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:54.164942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:26:54.165466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:54.165437    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:26:55.169461 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:55.169432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:26:55.170232 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:55.170210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:26:57.050418 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:57.050376    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:26:59.759902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:59.759868    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:26:59.760320 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:59.760132    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container" containerID="cri-o://842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8" gracePeriod=30
Sep 09 21:26:59.760320 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:26:59.760170    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kube-rbac-proxy" containerID="cri-o://358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749" gracePeriod=30
Sep 09 21:27:00.164957 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:00.164916    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 21:27:00.203847 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:00.203810    2581 generic.go:356] "Generic (PLEG): container finished" podID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerID="358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749" exitCode=2
Sep 09 21:27:00.204006 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:00.203881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerDied","Data":"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"}
Sep 09 21:27:01.057649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:01.057619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:27:01.057649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:01.057619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:27:01.947820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:01.947792    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:27:02.092658 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert\") pod \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") "
Sep 09 21:27:02.092658 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config\") pod \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") "
Sep 09 21:27:02.092658 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location\") pod \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") "
Sep 09 21:27:02.093330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p7ln\" (UniqueName: \"kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln\") pod \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") "
Sep 09 21:27:02.093330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") pod \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\" (UID: \"925de38a-3d5d-4dfa-98bf-7dcfd090c934\") "
Sep 09 21:27:02.093330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.092966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "925de38a-3d5d-4dfa-98bf-7dcfd090c934" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:02.093330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.093199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "925de38a-3d5d-4dfa-98bf-7dcfd090c934" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:02.093330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.093211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "925de38a-3d5d-4dfa-98bf-7dcfd090c934" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:02.094873 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.094855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln" (OuterVolumeSpecName: "kube-api-access-2p7ln") pod "925de38a-3d5d-4dfa-98bf-7dcfd090c934" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934"). InnerVolumeSpecName "kube-api-access-2p7ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:27:02.094944 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.094931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "925de38a-3d5d-4dfa-98bf-7dcfd090c934" (UID: "925de38a-3d5d-4dfa-98bf-7dcfd090c934"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:02.193200 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.193166    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925de38a-3d5d-4dfa-98bf-7dcfd090c934-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:02.193200 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.193197    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:02.193200 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.193207    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:02.193487 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.193216    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:02.193487 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.193226    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7ln\" (UniqueName: \"kubernetes.io/projected/925de38a-3d5d-4dfa-98bf-7dcfd090c934-kube-api-access-2p7ln\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:02.213454 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.213424    2581 generic.go:356] "Generic (PLEG): container finished" podID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerID="842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8" exitCode=0
Sep 09 21:27:02.213650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.213480    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"
Sep 09 21:27:02.213650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.213494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerDied","Data":"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"}
Sep 09 21:27:02.213650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.213554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s" event={"ID":"925de38a-3d5d-4dfa-98bf-7dcfd090c934","Type":"ContainerDied","Data":"1bc5e14effc3a2adef2967e16cf6f793c7bab1dff35c4d7e2e850666caad02f9"}
Sep 09 21:27:02.213650 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.213582    2581 scope.go:117] "RemoveContainer" containerID="358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"
Sep 09 21:27:02.221994 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.221974    2581 scope.go:117] "RemoveContainer" containerID="842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"
Sep 09 21:27:02.235833 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.235809    2581 scope.go:117] "RemoveContainer" containerID="c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"
Sep 09 21:27:02.241792 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.241760    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:27:02.244473 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.244443    2581 scope.go:117] "RemoveContainer" containerID="358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"
Sep 09 21:27:02.244886 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:02.244861    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749\": container with ID starting with 358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749 not found: ID does not exist" containerID="358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"
Sep 09 21:27:02.244973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.244898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749"} err="failed to get container status \"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749\": rpc error: code = NotFound desc = could not find container \"358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749\": container with ID starting with 358d780ee0f7eee2f894f16d13f9030d68043376725aceceb318d4279f6c3749 not found: ID does not exist"
Sep 09 21:27:02.244973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.244918    2581 scope.go:117] "RemoveContainer" containerID="842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"
Sep 09 21:27:02.245250 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:02.245232    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8\": container with ID starting with 842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8 not found: ID does not exist" containerID="842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"
Sep 09 21:27:02.245309 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.245253    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8"} err="failed to get container status \"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8\": rpc error: code = NotFound desc = could not find container \"842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8\": container with ID starting with 842ba03ce68d3e3519c839b5f4f80bae49c9766c6fae0ac24edfcb6b3fea99f8 not found: ID does not exist"
Sep 09 21:27:02.245309 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.245275    2581 scope.go:117] "RemoveContainer" containerID="c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"
Sep 09 21:27:02.245558 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:02.245534    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208\": container with ID starting with c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208 not found: ID does not exist" containerID="c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"
Sep 09 21:27:02.245612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.245566    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208"} err="failed to get container status \"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208\": rpc error: code = NotFound desc = could not find container \"c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208\": container with ID starting with c72755abdafadd76d118a522792c9f1b99ec8abad5f75605a87473e11a52c208 not found: ID does not exist"
Sep 09 21:27:02.246120 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:02.246106    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xn62s"]
Sep 09 21:27:03.030337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:03.030302    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" path="/var/lib/kubelet/pods/925de38a-3d5d-4dfa-98bf-7dcfd090c934/volumes"
Sep 09 21:27:04.165448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:04.165409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:27:05.019965 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.019929    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:27:05.020224 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020225    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020238    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="storage-initializer"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020243    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="storage-initializer"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020254    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="storage-initializer"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="storage-initializer"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020267    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kube-rbac-proxy"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020271    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kube-rbac-proxy"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020279    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020283    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020291    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container"
Sep 09 21:27:05.020295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020296    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container"
Sep 09 21:27:05.020869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020346    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kserve-container"
Sep 09 21:27:05.020869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020352    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kserve-container"
Sep 09 21:27:05.020869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020362    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49af8a63-7e2e-4e42-a6c2-87b0817cdcf1" containerName="kube-rbac-proxy"
Sep 09 21:27:05.020869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.020368    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="925de38a-3d5d-4dfa-98bf-7dcfd090c934" containerName="kube-rbac-proxy"
Sep 09 21:27:05.049398 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.049367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.052237 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.052208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 09 21:27:05.052440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.052332    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 21:27:05.056453 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.056376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:27:05.216289 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.216247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld8z\" (UniqueName: \"kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.216731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.216305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.216731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.216334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.216731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.216358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.216731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.216421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.317879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.317785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld8z\" (UniqueName: \"kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.317879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.317827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.317879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.317845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.317879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.317860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.317879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.317883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.318306 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.318281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.318530 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.318511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.318637 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.318619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.320404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.320375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.328075 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.328039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld8z\" (UniqueName: \"kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z\") pod \"isvc-xgboost-predictor-7566d65bf7-b5lvr\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.360163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.360119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:05.671850 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:05.671823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:27:05.673642 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:27:05.673614    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6280580d_ec06_4025_98d7_c25956d29073.slice/crio-c5aac35577514bb029aa40777aa85edfef076d3acf423e7f42d85c763e4303f0 WatchSource:0}: Error finding container c5aac35577514bb029aa40777aa85edfef076d3acf423e7f42d85c763e4303f0: Status 404 returned error can't find the container with id c5aac35577514bb029aa40777aa85edfef076d3acf423e7f42d85c763e4303f0
Sep 09 21:27:06.234231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:06.234194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerStarted","Data":"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"}
Sep 09 21:27:06.234719 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:06.234303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerStarted","Data":"c5aac35577514bb029aa40777aa85edfef076d3acf423e7f42d85c763e4303f0"}
Sep 09 21:27:07.049885 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:07.049800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:27:10.250182 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:10.250141    2581 generic.go:356] "Generic (PLEG): container finished" podID="6280580d-ec06-4025-98d7-c25956d29073" containerID="fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21" exitCode=0
Sep 09 21:27:10.250583 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:10.250204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerDied","Data":"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"}
Sep 09 21:27:10.251519 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:10.251502    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:27:14.165529 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:14.165487    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:27:17.050976 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:17.050935    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:27:24.165518 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:24.165474    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:27:27.050661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:27.050620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:27:29.339020 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:29.338980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerStarted","Data":"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"}
Sep 09 21:27:29.339400 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:29.339070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerStarted","Data":"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"}
Sep 09 21:27:29.339400 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:29.339310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:29.361561 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:29.361505    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podStartSLOduration=1.67122931 podStartE2EDuration="24.36149011s" totalImagesPullingTime="18.411576093s" totalInitContainerRuntime="4.278684707s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:29.359245606 +0000 UTC m=+2128.855519648" watchObservedRunningTime="2026-09-09 21:27:29.36149011 +0000 UTC m=+2128.857764209"
Sep 09 21:27:30.343370 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:30.343339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:30.344461 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:30.344432    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:27:31.347362 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:31.347320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:27:34.166167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:34.166128    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:27:36.351922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:36.351885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:27:36.352567 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:36.352536    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:27:37.050816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:37.050777    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 21:27:43.030825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:43.030797    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:27:44.165865 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:44.165834    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:27:44.295258 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:44.295222    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:27:44.397063 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:44.397022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container" containerID="cri-o://d10ac35323361f96d8587eec3467216b266ec40688084ed189b3dffeed7a6ec0" gracePeriod=30
Sep 09 21:27:44.397279 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:44.397051    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kube-rbac-proxy" containerID="cri-o://4d5dc34774eb465ff4118a2d015720ec08d24872ef3a6d613d90417bc95fdfd9" gracePeriod=30
Sep 09 21:27:45.402225 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:45.402195    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerID="4d5dc34774eb465ff4118a2d015720ec08d24872ef3a6d613d90417bc95fdfd9" exitCode=2
Sep 09 21:27:45.402603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:45.402230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerDied","Data":"4d5dc34774eb465ff4118a2d015720ec08d24872ef3a6d613d90417bc95fdfd9"}
Sep 09 21:27:46.352587 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:46.352548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:27:47.044777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:47.044734    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 09 21:27:49.419183 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.419150    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerID="d10ac35323361f96d8587eec3467216b266ec40688084ed189b3dffeed7a6ec0" exitCode=0
Sep 09 21:27:49.419570 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.419225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerDied","Data":"d10ac35323361f96d8587eec3467216b266ec40688084ed189b3dffeed7a6ec0"}
Sep 09 21:27:49.439520 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.439494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:27:49.590819 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.590723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config\") pod \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") "
Sep 09 21:27:49.590819 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.590759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b7bw\" (UniqueName: \"kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw\") pod \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") "
Sep 09 21:27:49.591048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.590828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") pod \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") "
Sep 09 21:27:49.591048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.590859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location\") pod \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") "
Sep 09 21:27:49.591048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.590907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert\") pod \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\" (UID: \"4ebf755b-c2ce-4bbd-a61e-674a0aff9667\") "
Sep 09 21:27:49.591215 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.591093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ebf755b-c2ce-4bbd-a61e-674a0aff9667" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:49.591276 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.591255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ebf755b-c2ce-4bbd-a61e-674a0aff9667" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:49.591329 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.591306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ebf755b-c2ce-4bbd-a61e-674a0aff9667" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:49.593117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.593089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ebf755b-c2ce-4bbd-a61e-674a0aff9667" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:49.593117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.593110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw" (OuterVolumeSpecName: "kube-api-access-4b7bw") pod "4ebf755b-c2ce-4bbd-a61e-674a0aff9667" (UID: "4ebf755b-c2ce-4bbd-a61e-674a0aff9667"). InnerVolumeSpecName "kube-api-access-4b7bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:27:49.691661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.691617    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:49.691661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.691652    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:49.691661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.691662    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:49.691661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.691671    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:49.691966 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:49.691680    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b7bw\" (UniqueName: \"kubernetes.io/projected/4ebf755b-c2ce-4bbd-a61e-674a0aff9667-kube-api-access-4b7bw\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:50.233816 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.233778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:27:50.234026 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.234004    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" containerID="cri-o://e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955" gracePeriod=30
Sep 09 21:27:50.234119 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.234065    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kube-rbac-proxy" containerID="cri-o://91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb" gracePeriod=30
Sep 09 21:27:50.388047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:27:50.388333 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388319    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container"
Sep 09 21:27:50.388382 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388335    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container"
Sep 09 21:27:50.388382 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="storage-initializer"
Sep 09 21:27:50.388382 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388356    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="storage-initializer"
Sep 09 21:27:50.388382 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388371    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kube-rbac-proxy"
Sep 09 21:27:50.388382 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388377    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kube-rbac-proxy"
Sep 09 21:27:50.388531 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388420    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kserve-container"
Sep 09 21:27:50.388531 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.388427    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" containerName="kube-rbac-proxy"
Sep 09 21:27:50.391436 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.391420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.406267 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.406236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:27:50.426027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.425997    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"
Sep 09 21:27:50.426480 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.426000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8" event={"ID":"4ebf755b-c2ce-4bbd-a61e-674a0aff9667","Type":"ContainerDied","Data":"a0590383d8d1fca96fd4ce1bd7183394c7f07db52a13090c5422059913f736ac"}
Sep 09 21:27:50.426480 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.426153    2581 scope.go:117] "RemoveContainer" containerID="4d5dc34774eb465ff4118a2d015720ec08d24872ef3a6d613d90417bc95fdfd9"
Sep 09 21:27:50.428597 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.428572    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerID="91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb" exitCode=2
Sep 09 21:27:50.428762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.428627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerDied","Data":"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"}
Sep 09 21:27:50.435735 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.435716    2581 scope.go:117] "RemoveContainer" containerID="d10ac35323361f96d8587eec3467216b266ec40688084ed189b3dffeed7a6ec0"
Sep 09 21:27:50.443968 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.443945    2581 scope.go:117] "RemoveContainer" containerID="0578a64c17dda60a3d9d89c944d15a0d11af9fc1a8860c50596785b65cfecf1d"
Sep 09 21:27:50.458985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.458953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:27:50.462440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.462411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-nsgt8"]
Sep 09 21:27:50.499202 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.499106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.499202 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.499155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgdd\" (UniqueName: \"kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.499400 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.499234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.499400 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.499265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.499400 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.499318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.600447 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.600413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.600447 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.600458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.600659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.600486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgdd\" (UniqueName: \"kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.600659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.600531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.600659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.600579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.601047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.601022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.601233 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.601214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.601281 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.601214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.602995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.602976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.610801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.610775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgdd\" (UniqueName: \"kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-pfjf4\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.701606 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.701566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:50.987267 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:50.987228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:27:50.990184 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:27:50.990148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe8e1f_3f19_48cd_b44e_7524b72a59d5.slice/crio-7f28223697d9e2fc6bd3933fbc4e17d8faaa439416bd3ac54bba8304e79141a1 WatchSource:0}: Error finding container 7f28223697d9e2fc6bd3933fbc4e17d8faaa439416bd3ac54bba8304e79141a1: Status 404 returned error can't find the container with id 7f28223697d9e2fc6bd3933fbc4e17d8faaa439416bd3ac54bba8304e79141a1
Sep 09 21:27:51.030856 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:51.030826    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ebf755b-c2ce-4bbd-a61e-674a0aff9667" path="/var/lib/kubelet/pods/4ebf755b-c2ce-4bbd-a61e-674a0aff9667/volumes"
Sep 09 21:27:51.435498 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:51.435464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerStarted","Data":"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"}
Sep 09 21:27:51.435922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:51.435525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerStarted","Data":"7f28223697d9e2fc6bd3933fbc4e17d8faaa439416bd3ac54bba8304e79141a1"}
Sep 09 21:27:54.161187 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:54.161094    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 09 21:27:54.166482 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:54.166453    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 09 21:27:55.212591 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.212560    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:27:55.336951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.336852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location\") pod \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") "
Sep 09 21:27:55.336951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.336924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert\") pod \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") "
Sep 09 21:27:55.336951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.336948    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config\") pod \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") "
Sep 09 21:27:55.337371 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.336969    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") pod \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") "
Sep 09 21:27:55.337371 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.336997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vl7z\" (UniqueName: \"kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z\") pod \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\" (UID: \"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625\") "
Sep 09 21:27:55.337371 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.337309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:55.337371 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.337322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:55.337526 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.337374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:55.339231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.339201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:55.339231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.339221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z" (OuterVolumeSpecName: "kube-api-access-9vl7z") pod "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" (UID: "b6ba385d-3ddd-4d3c-ae27-ffb64dad0625"). InnerVolumeSpecName "kube-api-access-9vl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:27:55.438087 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.438052    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:55.438087 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.438085    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:55.438335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.438098    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:55.438335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.438112    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:55.438335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.438124    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vl7z\" (UniqueName: \"kubernetes.io/projected/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625-kube-api-access-9vl7z\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:27:55.451853 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.451819    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerID="c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d" exitCode=0
Sep 09 21:27:55.452004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.451896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerDied","Data":"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"}
Sep 09 21:27:55.453941 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.453916    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerID="e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955" exitCode=0
Sep 09 21:27:55.454034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.453944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerDied","Data":"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"}
Sep 09 21:27:55.454034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.453975    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"
Sep 09 21:27:55.454034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.453998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95" event={"ID":"b6ba385d-3ddd-4d3c-ae27-ffb64dad0625","Type":"ContainerDied","Data":"8e8a95b7d80df0aed9d8e9865ec09a344bc5d27d92c8fae0014911e72c629976"}
Sep 09 21:27:55.454034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.454028    2581 scope.go:117] "RemoveContainer" containerID="91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"
Sep 09 21:27:55.462782 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.462755    2581 scope.go:117] "RemoveContainer" containerID="e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"
Sep 09 21:27:55.470249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.470227    2581 scope.go:117] "RemoveContainer" containerID="20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"
Sep 09 21:27:55.479165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.479142    2581 scope.go:117] "RemoveContainer" containerID="91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"
Sep 09 21:27:55.479471 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:55.479450    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb\": container with ID starting with 91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb not found: ID does not exist" containerID="91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"
Sep 09 21:27:55.479525 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.479480    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb"} err="failed to get container status \"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb\": rpc error: code = NotFound desc = could not find container \"91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb\": container with ID starting with 91dcc534ddcb3fede1287d8ac23ac0e4127b76ce2459a94c3999a3f539919ccb not found: ID does not exist"
Sep 09 21:27:55.479525 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.479498    2581 scope.go:117] "RemoveContainer" containerID="e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"
Sep 09 21:27:55.479796 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:55.479776    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955\": container with ID starting with e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955 not found: ID does not exist" containerID="e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"
Sep 09 21:27:55.479875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.479805    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955"} err="failed to get container status \"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955\": rpc error: code = NotFound desc = could not find container \"e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955\": container with ID starting with e3e7c20b655361d7b9d05c82e45e55dda11af0b4658bdc1ed26b39bb16b01955 not found: ID does not exist"
Sep 09 21:27:55.479875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.479826    2581 scope.go:117] "RemoveContainer" containerID="20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"
Sep 09 21:27:55.480073 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:27:55.480057    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8\": container with ID starting with 20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8 not found: ID does not exist" containerID="20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"
Sep 09 21:27:55.480113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.480077    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8"} err="failed to get container status \"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8\": rpc error: code = NotFound desc = could not find container \"20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8\": container with ID starting with 20640552ccd0c3e5c01685b3ae01b94ec2cd830846cc54570573d6fdba27e6b8 not found: ID does not exist"
Sep 09 21:27:55.491419 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.490907    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:27:55.497675 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:55.497641    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-wsf95"]
Sep 09 21:27:56.352800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:56.352749    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:27:56.460446 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:56.460408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerStarted","Data":"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"}
Sep 09 21:27:56.460652 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:56.460516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerStarted","Data":"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"}
Sep 09 21:27:56.460746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:56.460695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:56.481588 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:56.481533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podStartSLOduration=2.698794886 podStartE2EDuration="6.481510905s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782716019s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:56.481148793 +0000 UTC m=+2155.977422873" watchObservedRunningTime="2026-09-09 21:27:56.481510905 +0000 UTC m=+2155.977784948"
Sep 09 21:27:57.030900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:57.030865    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" path="/var/lib/kubelet/pods/b6ba385d-3ddd-4d3c-ae27-ffb64dad0625/volumes"
Sep 09 21:27:57.465299 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:57.465252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:27:57.466647 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:57.466618    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:27:58.469299 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:27:58.469253    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:03.474162 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:03.474132    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:28:03.476082 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:03.474606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:06.353185 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:06.353142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:28:13.475181 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:13.475139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:16.353135 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:16.353090    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:28:23.475055 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:23.475004    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:26.352454 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:26.352411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:28:33.475030 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:33.474986    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:36.353754 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:36.353720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:28:43.475495 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:43.475461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:28:45.073861 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:45.073824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:28:45.074248 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:45.074078    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" containerID="cri-o://5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c" gracePeriod=30
Sep 09 21:28:45.074248 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:45.074117    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kube-rbac-proxy" containerID="cri-o://d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1" gracePeriod=30
Sep 09 21:28:45.649069 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:45.649035    2581 generic.go:356] "Generic (PLEG): container finished" podID="6280580d-ec06-4025-98d7-c25956d29073" containerID="d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1" exitCode=2
Sep 09 21:28:45.649069 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:45.649081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerDied","Data":"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"}
Sep 09 21:28:46.348406 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:46.348362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 21:28:46.352746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:46.352714    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 21:28:48.916648 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.916620    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:28:48.955439 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location\") pod \"6280580d-ec06-4025-98d7-c25956d29073\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") "
Sep 09 21:28:48.955640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls\") pod \"6280580d-ec06-4025-98d7-c25956d29073\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") "
Sep 09 21:28:48.955640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config\") pod \"6280580d-ec06-4025-98d7-c25956d29073\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") "
Sep 09 21:28:48.955640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert\") pod \"6280580d-ec06-4025-98d7-c25956d29073\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") "
Sep 09 21:28:48.955640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qld8z\" (UniqueName: \"kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z\") pod \"6280580d-ec06-4025-98d7-c25956d29073\" (UID: \"6280580d-ec06-4025-98d7-c25956d29073\") "
Sep 09 21:28:48.955894 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955796    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6280580d-ec06-4025-98d7-c25956d29073" (UID: "6280580d-ec06-4025-98d7-c25956d29073"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:48.955958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6280580d-ec06-4025-98d7-c25956d29073" (UID: "6280580d-ec06-4025-98d7-c25956d29073"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:28:48.956029 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.955997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6280580d-ec06-4025-98d7-c25956d29073" (UID: "6280580d-ec06-4025-98d7-c25956d29073"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:28:48.957728 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.957688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z" (OuterVolumeSpecName: "kube-api-access-qld8z") pod "6280580d-ec06-4025-98d7-c25956d29073" (UID: "6280580d-ec06-4025-98d7-c25956d29073"). InnerVolumeSpecName "kube-api-access-qld8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:28:48.957841 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:48.957741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6280580d-ec06-4025-98d7-c25956d29073" (UID: "6280580d-ec06-4025-98d7-c25956d29073"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:28:49.056691 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.056659    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:28:49.056691 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.056687    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6280580d-ec06-4025-98d7-c25956d29073-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:28:49.056886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.056726    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qld8z\" (UniqueName: \"kubernetes.io/projected/6280580d-ec06-4025-98d7-c25956d29073-kube-api-access-qld8z\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:28:49.056886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.056740    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6280580d-ec06-4025-98d7-c25956d29073-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:28:49.056886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.056754    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6280580d-ec06-4025-98d7-c25956d29073-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:28:49.666018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.665982    2581 generic.go:356] "Generic (PLEG): container finished" podID="6280580d-ec06-4025-98d7-c25956d29073" containerID="5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c" exitCode=0
Sep 09 21:28:49.666215 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.666037    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"
Sep 09 21:28:49.666215 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.666040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerDied","Data":"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"}
Sep 09 21:28:49.666215 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.666165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr" event={"ID":"6280580d-ec06-4025-98d7-c25956d29073","Type":"ContainerDied","Data":"c5aac35577514bb029aa40777aa85edfef076d3acf423e7f42d85c763e4303f0"}
Sep 09 21:28:49.666215 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.666191    2581 scope.go:117] "RemoveContainer" containerID="d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"
Sep 09 21:28:49.674180 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.674161    2581 scope.go:117] "RemoveContainer" containerID="5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"
Sep 09 21:28:49.681913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.681887    2581 scope.go:117] "RemoveContainer" containerID="fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"
Sep 09 21:28:49.686266 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.686241    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:28:49.692242 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.692216    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-b5lvr"]
Sep 09 21:28:49.693026 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693003    2581 scope.go:117] "RemoveContainer" containerID="d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"
Sep 09 21:28:49.693341 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:28:49.693322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1\": container with ID starting with d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1 not found: ID does not exist" containerID="d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"
Sep 09 21:28:49.693408 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693348    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1"} err="failed to get container status \"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1\": rpc error: code = NotFound desc = could not find container \"d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1\": container with ID starting with d54157e4dacbed74d5f692a347f708075092c45f543ac2b25b051c639f1962e1 not found: ID does not exist"
Sep 09 21:28:49.693408 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693367    2581 scope.go:117] "RemoveContainer" containerID="5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"
Sep 09 21:28:49.693602 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:28:49.693586    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c\": container with ID starting with 5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c not found: ID does not exist" containerID="5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"
Sep 09 21:28:49.693638 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693606    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c"} err="failed to get container status \"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c\": rpc error: code = NotFound desc = could not find container \"5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c\": container with ID starting with 5ca35dc373650dd84e4bcfaa58072fe35cb2e859b765e221daac3a8bcdf6261c not found: ID does not exist"
Sep 09 21:28:49.693638 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693619    2581 scope.go:117] "RemoveContainer" containerID="fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"
Sep 09 21:28:49.693887 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:28:49.693869    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21\": container with ID starting with fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21 not found: ID does not exist" containerID="fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"
Sep 09 21:28:49.693946 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:49.693890    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21"} err="failed to get container status \"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21\": rpc error: code = NotFound desc = could not find container \"fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21\": container with ID starting with fdd99783427e682f149d890c4af4fd60e635040a80cf9f17d6b6640c81408f21 not found: ID does not exist"
Sep 09 21:28:51.029142 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:51.029107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6280580d-ec06-4025-98d7-c25956d29073" path="/var/lib/kubelet/pods/6280580d-ec06-4025-98d7-c25956d29073/volumes"
Sep 09 21:28:53.475490 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:28:53.475451    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:29:03.475543 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:03.475501    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:29:13.474897 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:13.474867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:29:20.390738 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:20.390682    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:29:20.391245 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:20.390955    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" containerID="cri-o://2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452" gracePeriod=30
Sep 09 21:29:20.391245 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:20.390994    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kube-rbac-proxy" containerID="cri-o://95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47" gracePeriod=30
Sep 09 21:29:20.783484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:20.783449    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerID="95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47" exitCode=2
Sep 09 21:29:20.783657 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:20.783512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerDied","Data":"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"}
Sep 09 21:29:23.469603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:23.469499    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 09 21:29:23.474934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:23.474895    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 09 21:29:25.439825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.439799    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:29:25.539151 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgdd\" (UniqueName: \"kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd\") pod \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") "
Sep 09 21:29:25.539151 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location\") pod \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") "
Sep 09 21:29:25.539151 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls\") pod \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") "
Sep 09 21:29:25.539151 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config\") pod \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") "
Sep 09 21:29:25.539488 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert\") pod \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\" (UID: \"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5\") "
Sep 09 21:29:25.539488 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" (UID: "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.539593 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" (UID: "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:25.539662 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539637    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.539727 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539673    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.539784 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.539763    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" (UID: "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:25.541469 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.541425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd" (OuterVolumeSpecName: "kube-api-access-vfgdd") pod "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" (UID: "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5"). InnerVolumeSpecName "kube-api-access-vfgdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:29:25.541606 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.541429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" (UID: "3cbe8e1f-3f19-48cd-b44e-7524b72a59d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:29:25.627815 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.627780    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:29:25.628066 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628053    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628076    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628081    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628094    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628103    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="storage-initializer"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628108    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="storage-initializer"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="storage-initializer"
Sep 09 21:29:25.628122 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628118    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="storage-initializer"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628134    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628141    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="storage-initializer"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628158    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="storage-initializer"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628164    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628169    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628210    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628219    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628225    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628230    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerName="kserve-container"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628235    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6280580d-ec06-4025-98d7-c25956d29073" containerName="kube-rbac-proxy"
Sep 09 21:29:25.628457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.628240    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ba385d-3ddd-4d3c-ae27-ffb64dad0625" containerName="kserve-container"
Sep 09 21:29:25.631364 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.631348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.640193 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.640171    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgdd\" (UniqueName: \"kubernetes.io/projected/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-kube-api-access-vfgdd\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.640310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.640218    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.640310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.640233    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.640761 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.640748    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 21:29:25.643071 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.643053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 21:29:25.678113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.678078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:29:25.723801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.723769    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:29:25.727055 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.727039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.730652 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.730630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 21:29:25.740930 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.740907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.741039 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.740955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.741039 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.740998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.741113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.741077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl22\" (UniqueName: \"kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.741148 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.741108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.747241 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.747220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:29:25.804725 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.804616    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" containerID="2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452" exitCode=0
Sep 09 21:29:25.804725 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.804674    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"
Sep 09 21:29:25.804913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.804715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerDied","Data":"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"}
Sep 09 21:29:25.804913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.804776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4" event={"ID":"3cbe8e1f-3f19-48cd-b44e-7524b72a59d5","Type":"ContainerDied","Data":"7f28223697d9e2fc6bd3933fbc4e17d8faaa439416bd3ac54bba8304e79141a1"}
Sep 09 21:29:25.804913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.804804    2581 scope.go:117] "RemoveContainer" containerID="95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"
Sep 09 21:29:25.813658 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.813640    2581 scope.go:117] "RemoveContainer" containerID="2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"
Sep 09 21:29:25.821417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.821390    2581 scope.go:117] "RemoveContainer" containerID="c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"
Sep 09 21:29:25.828599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.828579    2581 scope.go:117] "RemoveContainer" containerID="95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"
Sep 09 21:29:25.828913 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:29:25.828892    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47\": container with ID starting with 95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47 not found: ID does not exist" containerID="95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"
Sep 09 21:29:25.828993 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.828920    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47"} err="failed to get container status \"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47\": rpc error: code = NotFound desc = could not find container \"95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47\": container with ID starting with 95d44629c6754d7ef77677d524099cb6672eb5d5b7f606d13d3aa91bf72f9c47 not found: ID does not exist"
Sep 09 21:29:25.828993 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.828939    2581 scope.go:117] "RemoveContainer" containerID="2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"
Sep 09 21:29:25.829210 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:29:25.829192    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452\": container with ID starting with 2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452 not found: ID does not exist" containerID="2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"
Sep 09 21:29:25.829274 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.829222    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452"} err="failed to get container status \"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452\": rpc error: code = NotFound desc = could not find container \"2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452\": container with ID starting with 2656ebcdefb41266a83ed785cd4bd0a7f899d066476e08ec7268b2c93c3fb452 not found: ID does not exist"
Sep 09 21:29:25.829274 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.829247    2581 scope.go:117] "RemoveContainer" containerID="c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"
Sep 09 21:29:25.829611 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:29:25.829575    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d\": container with ID starting with c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d not found: ID does not exist" containerID="c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"
Sep 09 21:29:25.829729 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.829607    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d"} err="failed to get container status \"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d\": rpc error: code = NotFound desc = could not find container \"c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d\": container with ID starting with c49dadebcbb2a3cda90e4e46ea97f1e9697ee1c8481acdc1472f111a7f69967d not found: ID does not exist"
Sep 09 21:29:25.836471 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.836446    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:29:25.841612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.841580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl22\" (UniqueName: \"kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.841772 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.841629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khzb\" (UniqueName: \"kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.841772 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.841663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.841772 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.841693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.841971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.841945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.842085 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.842205 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.842205 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.842304 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.842304 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.842769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.842886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.842808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.843071 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.843050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.843887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.843863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-pfjf4"]
Sep 09 21:29:25.844583 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.844563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.850766 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.850743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl22\" (UniqueName: \"kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22\") pod \"isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.941301 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.941260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:25.943568 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.943538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.943755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.943573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.943755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.943592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.943755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.943628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khzb\" (UniqueName: \"kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.943755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.943653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.944045 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:29:25.943747    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 21:29:25.944045 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:29:25.943824    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls podName:976b98a2-3b7f-4660-b715-b8f2a9482f72 nodeName:}" failed. No retries permitted until 2026-09-09 21:29:26.443800771 +0000 UTC m=+2245.940074797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls") pod "isvc-canary-stop-predictor-59fbfd5bc-bs7kv" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 21:29:25.944166 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.944068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.944363 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.944340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.944427 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.944361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:25.952553 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:25.952494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khzb\" (UniqueName: \"kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:26.215673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.215637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:29:26.219304 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:29:26.219274    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e72010_ec86_467a_a65c_03759486d91b.slice/crio-744e5bd0fa2a8611b1f5b7a1dc11bcbe5974a30e7469b8acea3e99638f542c29 WatchSource:0}: Error finding container 744e5bd0fa2a8611b1f5b7a1dc11bcbe5974a30e7469b8acea3e99638f542c29: Status 404 returned error can't find the container with id 744e5bd0fa2a8611b1f5b7a1dc11bcbe5974a30e7469b8acea3e99638f542c29
Sep 09 21:29:26.447199 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.447160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:26.449590 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.449555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-bs7kv\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:26.636656 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.636566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:26.810229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.810188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerStarted","Data":"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"}
Sep 09 21:29:26.810414 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.810265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerStarted","Data":"744e5bd0fa2a8611b1f5b7a1dc11bcbe5974a30e7469b8acea3e99638f542c29"}
Sep 09 21:29:26.909524 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:26.909487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:29:26.912570 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:29:26.912540    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976b98a2_3b7f_4660_b715_b8f2a9482f72.slice/crio-bbb49cd12320643be50ced3625e03f0ff0cef2508835f4ad4ba3b2daac54ecf5 WatchSource:0}: Error finding container bbb49cd12320643be50ced3625e03f0ff0cef2508835f4ad4ba3b2daac54ecf5: Status 404 returned error can't find the container with id bbb49cd12320643be50ced3625e03f0ff0cef2508835f4ad4ba3b2daac54ecf5
Sep 09 21:29:27.030361 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:27.030330    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cbe8e1f-3f19-48cd-b44e-7524b72a59d5" path="/var/lib/kubelet/pods/3cbe8e1f-3f19-48cd-b44e-7524b72a59d5/volumes"
Sep 09 21:29:27.816557 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:27.816519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerStarted","Data":"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"}
Sep 09 21:29:27.816986 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:27.816615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerStarted","Data":"bbb49cd12320643be50ced3625e03f0ff0cef2508835f4ad4ba3b2daac54ecf5"}
Sep 09 21:29:30.830236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:30.830202    2581 generic.go:356] "Generic (PLEG): container finished" podID="45e72010-ec86-467a-a65c-03759486d91b" containerID="6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25" exitCode=0
Sep 09 21:29:30.830661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:30.830275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerDied","Data":"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"}
Sep 09 21:29:31.836492 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.836452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerStarted","Data":"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"}
Sep 09 21:29:31.836989 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.836526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerStarted","Data":"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"}
Sep 09 21:29:31.836989 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.836848    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:31.837965 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.837942    2581 generic.go:356] "Generic (PLEG): container finished" podID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerID="00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956" exitCode=0
Sep 09 21:29:31.838073 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.838016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerDied","Data":"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"}
Sep 09 21:29:31.862373 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:31.862329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podStartSLOduration=3.120111753 podStartE2EDuration="6.862312874s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742201121s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:31.861059591 +0000 UTC m=+2251.357333666" watchObservedRunningTime="2026-09-09 21:29:31.862312874 +0000 UTC m=+2251.358586915"
Sep 09 21:29:32.843855 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.843814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerStarted","Data":"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"}
Sep 09 21:29:32.844283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.843924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerStarted","Data":"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"}
Sep 09 21:29:32.844283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.844103    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:32.844283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.844238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:32.845431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.845404    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:29:32.865656 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:32.865605    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podStartSLOduration=3.895771784 podStartE2EDuration="7.865591712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.969819928s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:32.864193909 +0000 UTC m=+2252.360467951" watchObservedRunningTime="2026-09-09 21:29:32.865591712 +0000 UTC m=+2252.361865754"
Sep 09 21:29:33.848326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:33.848285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:29:33.848326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:33.848325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:33.849383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:33.849360    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:29:34.851693 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:34.851649    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:29:38.852788 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:38.852755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:29:38.853325 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:38.853287    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:29:39.856756 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:39.856718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:29:39.857204 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:39.857177    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:29:48.853273 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:48.853233    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:29:49.857546 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:49.857505    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:29:58.853543 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:58.853505    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:29:59.857251 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:29:59.857214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:30:08.853886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:08.853844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:30:09.857176 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:09.857136    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:30:18.853965 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:18.853923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:30:19.858086 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:19.858045    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:30:28.853428 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:28.853389    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:30:29.857178 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:29.857138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:30:38.853291 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:38.853248    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:30:39.857753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:39.857716    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 21:30:40.911786 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:40.911748    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:30:40.915234 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:40.915215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:40.918039 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:40.918010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 09 21:30:40.918185 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:40.918051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:30:40.927142 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:40.927105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:30:41.040442 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.040409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.040442 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.040448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.040663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.040476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmq2\" (UniqueName: \"kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.040663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.040504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.040663 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.040524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141280 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmq2\" (UniqueName: \"kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141481 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141481 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141481 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141481 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.141682 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:30:41.141494    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 09 21:30:41.141682 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:30:41.141568    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls podName:a106433b-890a-4dd1-af0d-d38b1a8aff8b nodeName:}" failed. No retries permitted until 2026-09-09 21:30:41.641547954 +0000 UTC m=+2321.137821982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-t86sg" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 09 21:30:41.141878 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.141859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.142061 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.142045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.142139 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.142116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.153130 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.153096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmq2\" (UniqueName: \"kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.644951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.644918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.647320 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.647286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-t86sg\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:41.826687 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:41.826641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:42.099538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:42.099416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:30:42.103206 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:30:42.103174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda106433b_890a_4dd1_af0d_d38b1a8aff8b.slice/crio-6435bb5aa57b7de38456e93e3c40e1537209c8d4a20aa9f0aa35d39660219cb2 WatchSource:0}: Error finding container 6435bb5aa57b7de38456e93e3c40e1537209c8d4a20aa9f0aa35d39660219cb2: Status 404 returned error can't find the container with id 6435bb5aa57b7de38456e93e3c40e1537209c8d4a20aa9f0aa35d39660219cb2
Sep 09 21:30:42.116729 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:42.116684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerStarted","Data":"6435bb5aa57b7de38456e93e3c40e1537209c8d4a20aa9f0aa35d39660219cb2"}
Sep 09 21:30:43.122255 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:43.122208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerStarted","Data":"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"}
Sep 09 21:30:47.139637 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:47.139603    2581 generic.go:356] "Generic (PLEG): container finished" podID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerID="608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc" exitCode=0
Sep 09 21:30:47.140060 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:47.139689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerDied","Data":"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"}
Sep 09 21:30:48.146189 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.146156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerStarted","Data":"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"}
Sep 09 21:30:48.146572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.146216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerStarted","Data":"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"}
Sep 09 21:30:48.146572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.146403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:48.170927 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.170872    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podStartSLOduration=4.180648093 podStartE2EDuration="8.170853475s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990205382s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:30:48.169205148 +0000 UTC m=+2327.665479203" watchObservedRunningTime="2026-09-09 21:30:48.170853475 +0000 UTC m=+2327.667127516"
Sep 09 21:30:48.853866 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.853836    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:30:48.997446 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.997407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:30:48.997764 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.997712    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container" containerID="cri-o://1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc" gracePeriod=30
Sep 09 21:30:48.997942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.997819    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kube-rbac-proxy" containerID="cri-o://a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08" gracePeriod=30
Sep 09 21:30:48.998642 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:48.998623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:30:49.152822 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:49.152784    2581 generic.go:356] "Generic (PLEG): container finished" podID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerID="a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08" exitCode=2
Sep 09 21:30:49.153197 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:49.152858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerDied","Data":"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"}
Sep 09 21:30:49.153264 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:49.153203    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:49.154605 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:49.154577    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:30:49.852404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:49.852359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 09 21:30:50.156857 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:50.156817    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:30:54.045814 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.045789    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:30:54.149606 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.149571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") pod \"976b98a2-3b7f-4660-b715-b8f2a9482f72\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") "
Sep 09 21:30:54.149800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.149639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location\") pod \"976b98a2-3b7f-4660-b715-b8f2a9482f72\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") "
Sep 09 21:30:54.149800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.149678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2khzb\" (UniqueName: \"kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb\") pod \"976b98a2-3b7f-4660-b715-b8f2a9482f72\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") "
Sep 09 21:30:54.149800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.149693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config\") pod \"976b98a2-3b7f-4660-b715-b8f2a9482f72\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") "
Sep 09 21:30:54.149800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.149735    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert\") pod \"976b98a2-3b7f-4660-b715-b8f2a9482f72\" (UID: \"976b98a2-3b7f-4660-b715-b8f2a9482f72\") "
Sep 09 21:30:54.150079 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.150050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "976b98a2-3b7f-4660-b715-b8f2a9482f72" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:54.150160 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.150114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "976b98a2-3b7f-4660-b715-b8f2a9482f72" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:54.150217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.150194    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "976b98a2-3b7f-4660-b715-b8f2a9482f72" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:54.151825 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.151804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb" (OuterVolumeSpecName: "kube-api-access-2khzb") pod "976b98a2-3b7f-4660-b715-b8f2a9482f72" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72"). InnerVolumeSpecName "kube-api-access-2khzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:30:54.151947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.151852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "976b98a2-3b7f-4660-b715-b8f2a9482f72" (UID: "976b98a2-3b7f-4660-b715-b8f2a9482f72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:30:54.173491 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.173457    2581 generic.go:356] "Generic (PLEG): container finished" podID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerID="1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc" exitCode=0
Sep 09 21:30:54.173627 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.173510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerDied","Data":"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"}
Sep 09 21:30:54.173627 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.173523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"
Sep 09 21:30:54.173627 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.173607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv" event={"ID":"976b98a2-3b7f-4660-b715-b8f2a9482f72","Type":"ContainerDied","Data":"bbb49cd12320643be50ced3625e03f0ff0cef2508835f4ad4ba3b2daac54ecf5"}
Sep 09 21:30:54.173770 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.173646    2581 scope.go:117] "RemoveContainer" containerID="a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"
Sep 09 21:30:54.182139 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.182117    2581 scope.go:117] "RemoveContainer" containerID="1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"
Sep 09 21:30:54.189806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.189757    2581 scope.go:117] "RemoveContainer" containerID="00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"
Sep 09 21:30:54.195528 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.195498    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:30:54.198467 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.198447    2581 scope.go:117] "RemoveContainer" containerID="a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"
Sep 09 21:30:54.198806 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:30:54.198781    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08\": container with ID starting with a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08 not found: ID does not exist" containerID="a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"
Sep 09 21:30:54.198889 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.198817    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08"} err="failed to get container status \"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08\": rpc error: code = NotFound desc = could not find container \"a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08\": container with ID starting with a5da54cb687168023f22f08802c8eb277a41d72584ea3b28a04d5aca5e4b5e08 not found: ID does not exist"
Sep 09 21:30:54.198889 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.198836    2581 scope.go:117] "RemoveContainer" containerID="1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"
Sep 09 21:30:54.199132 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:30:54.199114    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc\": container with ID starting with 1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc not found: ID does not exist" containerID="1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"
Sep 09 21:30:54.199180 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.199140    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc"} err="failed to get container status \"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc\": rpc error: code = NotFound desc = could not find container \"1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc\": container with ID starting with 1539eb15917c718c1d2814e626f3d2fb2dd0dc8c0885d9cc6ec87be51d09d8fc not found: ID does not exist"
Sep 09 21:30:54.199180 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.199155    2581 scope.go:117] "RemoveContainer" containerID="00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"
Sep 09 21:30:54.199253 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.199229    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-bs7kv"]
Sep 09 21:30:54.199398 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:30:54.199381    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956\": container with ID starting with 00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956 not found: ID does not exist" containerID="00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"
Sep 09 21:30:54.199440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.199408    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956"} err="failed to get container status \"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956\": rpc error: code = NotFound desc = could not find container \"00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956\": container with ID starting with 00e14debeb0e9c3d63971fa109e7ee3b77ed3c72418bf1fefab446ab5577d956 not found: ID does not exist"
Sep 09 21:30:54.250973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.250933    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976b98a2-3b7f-4660-b715-b8f2a9482f72-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:30:54.250973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.250966    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2khzb\" (UniqueName: \"kubernetes.io/projected/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-api-access-2khzb\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:30:54.250973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.250976    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:30:54.251213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.250985    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976b98a2-3b7f-4660-b715-b8f2a9482f72-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:30:54.251213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:54.250998    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976b98a2-3b7f-4660-b715-b8f2a9482f72-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:30:55.031101 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.031069    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" path="/var/lib/kubelet/pods/976b98a2-3b7f-4660-b715-b8f2a9482f72/volumes"
Sep 09 21:30:55.161649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.161622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:30:55.162196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.162168    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:30:55.611647 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.611609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:30:55.611938 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.611911    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" containerID="cri-o://9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154" gracePeriod=30
Sep 09 21:30:55.612030 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:55.611956    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kube-rbac-proxy" containerID="cri-o://eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604" gracePeriod=30
Sep 09 21:30:56.184801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:56.184765    2581 generic.go:356] "Generic (PLEG): container finished" podID="45e72010-ec86-467a-a65c-03759486d91b" containerID="eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604" exitCode=2
Sep 09 21:30:56.185180 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:56.184839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerDied","Data":"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"}
Sep 09 21:30:58.848995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:58.848951    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 09 21:30:58.853329 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:30:58.853303    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 09 21:31:00.853003 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.852980    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:31:00.900104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert\") pod \"45e72010-ec86-467a-a65c-03759486d91b\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") "
Sep 09 21:31:00.900313 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggl22\" (UniqueName: \"kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22\") pod \"45e72010-ec86-467a-a65c-03759486d91b\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") "
Sep 09 21:31:00.900313 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location\") pod \"45e72010-ec86-467a-a65c-03759486d91b\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") "
Sep 09 21:31:00.900313 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config\") pod \"45e72010-ec86-467a-a65c-03759486d91b\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") "
Sep 09 21:31:00.900313 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls\") pod \"45e72010-ec86-467a-a65c-03759486d91b\" (UID: \"45e72010-ec86-467a-a65c-03759486d91b\") "
Sep 09 21:31:00.900550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900483    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45e72010-ec86-467a-a65c-03759486d91b" (UID: "45e72010-ec86-467a-a65c-03759486d91b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:31:00.900550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900533    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45e72010-ec86-467a-a65c-03759486d91b" (UID: "45e72010-ec86-467a-a65c-03759486d91b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:00.900636 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.900543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45e72010-ec86-467a-a65c-03759486d91b" (UID: "45e72010-ec86-467a-a65c-03759486d91b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:31:00.902326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.902301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45e72010-ec86-467a-a65c-03759486d91b" (UID: "45e72010-ec86-467a-a65c-03759486d91b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:00.902447 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:00.902424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22" (OuterVolumeSpecName: "kube-api-access-ggl22") pod "45e72010-ec86-467a-a65c-03759486d91b" (UID: "45e72010-ec86-467a-a65c-03759486d91b"). InnerVolumeSpecName "kube-api-access-ggl22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:01.001673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.001637    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggl22\" (UniqueName: \"kubernetes.io/projected/45e72010-ec86-467a-a65c-03759486d91b-kube-api-access-ggl22\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:31:01.001673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.001666    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45e72010-ec86-467a-a65c-03759486d91b-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:31:01.001673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.001676    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:31:01.001951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.001689    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45e72010-ec86-467a-a65c-03759486d91b-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:31:01.001951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.001718    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45e72010-ec86-467a-a65c-03759486d91b-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:31:01.206598 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.206502    2581 generic.go:356] "Generic (PLEG): container finished" podID="45e72010-ec86-467a-a65c-03759486d91b" containerID="9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154" exitCode=0
Sep 09 21:31:01.206598 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.206558    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"
Sep 09 21:31:01.206598 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.206575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerDied","Data":"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"}
Sep 09 21:31:01.206871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.206633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq" event={"ID":"45e72010-ec86-467a-a65c-03759486d91b","Type":"ContainerDied","Data":"744e5bd0fa2a8611b1f5b7a1dc11bcbe5974a30e7469b8acea3e99638f542c29"}
Sep 09 21:31:01.206871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.206659    2581 scope.go:117] "RemoveContainer" containerID="eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"
Sep 09 21:31:01.214550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.214528    2581 scope.go:117] "RemoveContainer" containerID="9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"
Sep 09 21:31:01.221992 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.221970    2581 scope.go:117] "RemoveContainer" containerID="6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"
Sep 09 21:31:01.227632 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.227594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:31:01.229458 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.229433    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5bb8f495f4-gpwwq"]
Sep 09 21:31:01.230415 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.230398    2581 scope.go:117] "RemoveContainer" containerID="eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"
Sep 09 21:31:01.230749 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:31:01.230732    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604\": container with ID starting with eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604 not found: ID does not exist" containerID="eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"
Sep 09 21:31:01.230810 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.230758    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604"} err="failed to get container status \"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604\": rpc error: code = NotFound desc = could not find container \"eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604\": container with ID starting with eedb1703960f6ebf59da0abe32913fd00609f66055f4a796e7d74eba26631604 not found: ID does not exist"
Sep 09 21:31:01.230810 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.230776    2581 scope.go:117] "RemoveContainer" containerID="9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"
Sep 09 21:31:01.231061 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:31:01.231042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154\": container with ID starting with 9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154 not found: ID does not exist" containerID="9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"
Sep 09 21:31:01.231112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.231066    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154"} err="failed to get container status \"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154\": rpc error: code = NotFound desc = could not find container \"9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154\": container with ID starting with 9d2e44d5a6a232fb6b8ed6cf64dda00af93578c42a4fbb37d134b7c5694a1154 not found: ID does not exist"
Sep 09 21:31:01.231112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.231081    2581 scope.go:117] "RemoveContainer" containerID="6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"
Sep 09 21:31:01.231315 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:31:01.231297    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25\": container with ID starting with 6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25 not found: ID does not exist" containerID="6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"
Sep 09 21:31:01.231352 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:01.231321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25"} err="failed to get container status \"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25\": rpc error: code = NotFound desc = could not find container \"6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25\": container with ID starting with 6bb01031957f1a173f74afcd3cbac2d67a444f4900b727b4d5924d129fa85d25 not found: ID does not exist"
Sep 09 21:31:03.029728 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:03.029672    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e72010-ec86-467a-a65c-03759486d91b" path="/var/lib/kubelet/pods/45e72010-ec86-467a-a65c-03759486d91b/volumes"
Sep 09 21:31:05.162652 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.162613    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:31:05.961973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.961935    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:31:05.962302 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962286    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="storage-initializer"
Sep 09 21:31:05.962369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="storage-initializer"
Sep 09 21:31:05.962369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962326    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container"
Sep 09 21:31:05.962369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962335    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container"
Sep 09 21:31:05.962369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962353    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="storage-initializer"
Sep 09 21:31:05.962369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962362    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="storage-initializer"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962376    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962407    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962469    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kserve-container"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962479    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962490    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="976b98a2-3b7f-4660-b715-b8f2a9482f72" containerName="kube-rbac-proxy"
Sep 09 21:31:05.962624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.962500    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e72010-ec86-467a-a65c-03759486d91b" containerName="kserve-container"
Sep 09 21:31:05.967454 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.967430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:05.971440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.971418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 09 21:31:05.972503 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.972486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 21:31:05.978427 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:05.978401    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:31:06.038828 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.038789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.038998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.038914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.038998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.038960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.038998 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.038984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6wg\" (UniqueName: \"kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.039106 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.039008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6wg\" (UniqueName: \"kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140553 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140553 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140553 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.140553 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:31:06.140407    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 09 21:31:06.140553 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:31:06.140551    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls podName:6731bde2-60ee-4839-8972-1656fa360ecb nodeName:}" failed. No retries permitted until 2026-09-09 21:31:06.640529319 +0000 UTC m=+2346.136803359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-skk2b" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 09 21:31:06.140886 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.140861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.141098 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.141075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.141172 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.141136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.150750 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.150724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6wg\" (UniqueName: \"kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.645484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.645437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.647987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.647960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-skk2b\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:06.878619 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:06.878573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:07.157059 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:07.157032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:31:07.159116 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:31:07.159088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6731bde2_60ee_4839_8972_1656fa360ecb.slice/crio-1848a55e591d699e8ad49926681d943cfe5d10f55d577a41fe85465a3d96fb98 WatchSource:0}: Error finding container 1848a55e591d699e8ad49926681d943cfe5d10f55d577a41fe85465a3d96fb98: Status 404 returned error can't find the container with id 1848a55e591d699e8ad49926681d943cfe5d10f55d577a41fe85465a3d96fb98
Sep 09 21:31:07.233251 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:07.233217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerStarted","Data":"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"}
Sep 09 21:31:07.233443 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:07.233271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerStarted","Data":"1848a55e591d699e8ad49926681d943cfe5d10f55d577a41fe85465a3d96fb98"}
Sep 09 21:31:11.248380 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:11.248346    2581 generic.go:356] "Generic (PLEG): container finished" podID="6731bde2-60ee-4839-8972-1656fa360ecb" containerID="ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68" exitCode=0
Sep 09 21:31:11.248380 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:11.248384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerDied","Data":"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"}
Sep 09 21:31:15.162383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:15.162335    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:31:18.280370 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:18.280330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerStarted","Data":"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"}
Sep 09 21:31:18.280777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:18.280430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerStarted","Data":"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"}
Sep 09 21:31:18.280777 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:18.280634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:18.300664 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:18.300608    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podStartSLOduration=3.457207763 podStartE2EDuration="13.300592838s" totalImagesPullingTime="6.048600866s" totalInitContainerRuntime="3.794784209s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:18.299284183 +0000 UTC m=+2357.795558271" watchObservedRunningTime="2026-09-09 21:31:18.300592838 +0000 UTC m=+2357.796866880"
Sep 09 21:31:19.284332 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:19.284305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:19.285725 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:19.285671    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:31:20.288124 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:20.288082    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:31:25.162295 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:25.162252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:31:25.293165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:25.293133    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:31:25.293847 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:25.293817    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:31:35.162184 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:35.162147    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:31:35.293773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:35.293733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:31:45.162103 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:45.162060    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 21:31:45.294518 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:45.294474    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:31:55.163272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:55.163237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:31:55.293910 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:31:55.293873    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:32:00.982239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:00.982199    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:32:00.982736 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:00.982513    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container" containerID="cri-o://f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc" gracePeriod=30
Sep 09 21:32:00.982736 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:00.982576    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kube-rbac-proxy" containerID="cri-o://78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede" gracePeriod=30
Sep 09 21:32:01.080941 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:01.080903    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:32:01.082272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:01.082250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:32:01.443629 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:01.443596    2581 generic.go:356] "Generic (PLEG): container finished" podID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerID="78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede" exitCode=2
Sep 09 21:32:01.443827 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:01.443668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerDied","Data":"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"}
Sep 09 21:32:04.623959 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.623933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:32:04.714762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.714636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location\") pod \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") "
Sep 09 21:32:04.714762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.714677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") pod \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") "
Sep 09 21:32:04.714762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.714750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert\") pod \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") "
Sep 09 21:32:04.715056 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.714787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config\") pod \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") "
Sep 09 21:32:04.715056 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.714812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqmq2\" (UniqueName: \"kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2\") pod \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\" (UID: \"a106433b-890a-4dd1-af0d-d38b1a8aff8b\") "
Sep 09 21:32:04.715167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.715114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a106433b-890a-4dd1-af0d-d38b1a8aff8b" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:04.715167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.715116    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a106433b-890a-4dd1-af0d-d38b1a8aff8b" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:04.715167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.715149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a106433b-890a-4dd1-af0d-d38b1a8aff8b" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:04.716901 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.716871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2" (OuterVolumeSpecName: "kube-api-access-zqmq2") pod "a106433b-890a-4dd1-af0d-d38b1a8aff8b" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b"). InnerVolumeSpecName "kube-api-access-zqmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:04.716901 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.716888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a106433b-890a-4dd1-af0d-d38b1a8aff8b" (UID: "a106433b-890a-4dd1-af0d-d38b1a8aff8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:04.815381 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.815339    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:04.815381 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.815374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:04.815381 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.815384    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmq2\" (UniqueName: \"kubernetes.io/projected/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kube-api-access-zqmq2\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:04.815381 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.815392    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a106433b-890a-4dd1-af0d-d38b1a8aff8b-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:04.815654 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:04.815403    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a106433b-890a-4dd1-af0d-d38b1a8aff8b-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:05.294219 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.294128    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:32:05.459752 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.459676    2581 generic.go:356] "Generic (PLEG): container finished" podID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerID="f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc" exitCode=0
Sep 09 21:32:05.459752 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.459740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerDied","Data":"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"}
Sep 09 21:32:05.460004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.459757    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"
Sep 09 21:32:05.460004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.459813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg" event={"ID":"a106433b-890a-4dd1-af0d-d38b1a8aff8b","Type":"ContainerDied","Data":"6435bb5aa57b7de38456e93e3c40e1537209c8d4a20aa9f0aa35d39660219cb2"}
Sep 09 21:32:05.460004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.459845    2581 scope.go:117] "RemoveContainer" containerID="78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"
Sep 09 21:32:05.468254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.468232    2581 scope.go:117] "RemoveContainer" containerID="f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"
Sep 09 21:32:05.475505 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.475486    2581 scope.go:117] "RemoveContainer" containerID="608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"
Sep 09 21:32:05.479732 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.479689    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:32:05.483543 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.483525    2581 scope.go:117] "RemoveContainer" containerID="78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"
Sep 09 21:32:05.483887 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:05.483863    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede\": container with ID starting with 78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede not found: ID does not exist" containerID="78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"
Sep 09 21:32:05.483994 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.483894    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede"} err="failed to get container status \"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede\": rpc error: code = NotFound desc = could not find container \"78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede\": container with ID starting with 78c4bebc3114c44a7b39bbf7a3784e10ed443ce00225cedb43962211c38edede not found: ID does not exist"
Sep 09 21:32:05.483994 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.483913    2581 scope.go:117] "RemoveContainer" containerID="f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"
Sep 09 21:32:05.484212 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:05.484193    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc\": container with ID starting with f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc not found: ID does not exist" containerID="f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"
Sep 09 21:32:05.484277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.484222    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc"} err="failed to get container status \"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc\": rpc error: code = NotFound desc = could not find container \"f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc\": container with ID starting with f858e4224e41db3abe4fafc165f0832b9b9daeee60757cbc859ca530143d29dc not found: ID does not exist"
Sep 09 21:32:05.484277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.484245    2581 scope.go:117] "RemoveContainer" containerID="608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"
Sep 09 21:32:05.484489 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:05.484473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc\": container with ID starting with 608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc not found: ID does not exist" containerID="608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"
Sep 09 21:32:05.484542 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.484494    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc"} err="failed to get container status \"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc\": rpc error: code = NotFound desc = could not find container \"608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc\": container with ID starting with 608bc84f5204e1106bebea553a70cfbf4e778b581a5798a8f1170d41b4e9bffc not found: ID does not exist"
Sep 09 21:32:05.486645 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:05.486620    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-t86sg"]
Sep 09 21:32:07.029429 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:07.029337    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" path="/var/lib/kubelet/pods/a106433b-890a-4dd1-af0d-d38b1a8aff8b/volumes"
Sep 09 21:32:15.293842 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:15.293801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:32:25.294691 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:25.294602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:32:26.025457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:26.025413    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 09 21:32:36.026809 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:36.026774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:32:46.347679 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:46.347638    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:32:46.348271 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:46.347917    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container" containerID="cri-o://2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54" gracePeriod=30
Sep 09 21:32:46.348271 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:46.347956    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kube-rbac-proxy" containerID="cri-o://fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d" gracePeriod=30
Sep 09 21:32:46.611485 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:46.611400    2581 generic.go:356] "Generic (PLEG): container finished" podID="6731bde2-60ee-4839-8972-1656fa360ecb" containerID="fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d" exitCode=2
Sep 09 21:32:46.611485 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:46.611468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerDied","Data":"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"}
Sep 09 21:32:50.289319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.289274    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 09 21:32:50.887309 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.887282    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:32:50.961665 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.961629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location\") pod \"6731bde2-60ee-4839-8972-1656fa360ecb\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") "
Sep 09 21:32:50.961665 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.961669    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config\") pod \"6731bde2-60ee-4839-8972-1656fa360ecb\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") "
Sep 09 21:32:50.961940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.961726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert\") pod \"6731bde2-60ee-4839-8972-1656fa360ecb\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") "
Sep 09 21:32:50.961940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.961755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") pod \"6731bde2-60ee-4839-8972-1656fa360ecb\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") "
Sep 09 21:32:50.961940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.961771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh6wg\" (UniqueName: \"kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg\") pod \"6731bde2-60ee-4839-8972-1656fa360ecb\" (UID: \"6731bde2-60ee-4839-8972-1656fa360ecb\") "
Sep 09 21:32:50.962115 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.962086    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6731bde2-60ee-4839-8972-1656fa360ecb" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:50.962163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.962119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6731bde2-60ee-4839-8972-1656fa360ecb" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:50.962163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.962151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6731bde2-60ee-4839-8972-1656fa360ecb" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:50.963957 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.963935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg" (OuterVolumeSpecName: "kube-api-access-lh6wg") pod "6731bde2-60ee-4839-8972-1656fa360ecb" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb"). InnerVolumeSpecName "kube-api-access-lh6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:50.964102 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:50.964087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6731bde2-60ee-4839-8972-1656fa360ecb" (UID: "6731bde2-60ee-4839-8972-1656fa360ecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:51.062952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.062918    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6731bde2-60ee-4839-8972-1656fa360ecb-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:51.062952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.062947    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:51.062952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.062958    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6731bde2-60ee-4839-8972-1656fa360ecb-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:51.063172 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.062969    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6731bde2-60ee-4839-8972-1656fa360ecb-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:51.063172 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.062990    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6wg\" (UniqueName: \"kubernetes.io/projected/6731bde2-60ee-4839-8972-1656fa360ecb-kube-api-access-lh6wg\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:32:51.571539 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571507    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571785    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="storage-initializer"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571797    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="storage-initializer"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571808    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571813    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571822    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571840    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="storage-initializer"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571845    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="storage-initializer"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571859    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571866    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571872    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571910    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kserve-container"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571916    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571925    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kube-rbac-proxy"
Sep 09 21:32:51.571922 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.571931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a106433b-890a-4dd1-af0d-d38b1a8aff8b" containerName="kserve-container"
Sep 09 21:32:51.575138 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.575122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.577947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.577926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 09 21:32:51.578084 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.577929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:32:51.588595 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.588566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:32:51.631915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.631882    2581 generic.go:356] "Generic (PLEG): container finished" podID="6731bde2-60ee-4839-8972-1656fa360ecb" containerID="2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54" exitCode=0
Sep 09 21:32:51.632080 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.631935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerDied","Data":"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"}
Sep 09 21:32:51.632080 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.631983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b" event={"ID":"6731bde2-60ee-4839-8972-1656fa360ecb","Type":"ContainerDied","Data":"1848a55e591d699e8ad49926681d943cfe5d10f55d577a41fe85465a3d96fb98"}
Sep 09 21:32:51.632080 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.632007    2581 scope.go:117] "RemoveContainer" containerID="fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"
Sep 09 21:32:51.632080 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.631938    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"
Sep 09 21:32:51.640103 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.640084    2581 scope.go:117] "RemoveContainer" containerID="2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"
Sep 09 21:32:51.647509 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.647482    2581 scope.go:117] "RemoveContainer" containerID="ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"
Sep 09 21:32:51.654890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.654845    2581 scope.go:117] "RemoveContainer" containerID="fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"
Sep 09 21:32:51.655146 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:51.655127    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d\": container with ID starting with fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d not found: ID does not exist" containerID="fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"
Sep 09 21:32:51.655193 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.655158    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d"} err="failed to get container status \"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d\": rpc error: code = NotFound desc = could not find container \"fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d\": container with ID starting with fb23932be136708a2c02467d8504c4a09a0e201511f8f158233c20c057b9898d not found: ID does not exist"
Sep 09 21:32:51.655193 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.655178    2581 scope.go:117] "RemoveContainer" containerID="2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"
Sep 09 21:32:51.655447 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:51.655426    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54\": container with ID starting with 2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54 not found: ID does not exist" containerID="2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"
Sep 09 21:32:51.655544 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.655451    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54"} err="failed to get container status \"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54\": rpc error: code = NotFound desc = could not find container \"2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54\": container with ID starting with 2a86c896a0ff440c4d1826a678834773ed57a61bcc157f07ec0506ecf0046c54 not found: ID does not exist"
Sep 09 21:32:51.655544 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.655465    2581 scope.go:117] "RemoveContainer" containerID="ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"
Sep 09 21:32:51.655829 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:51.655800    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68\": container with ID starting with ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68 not found: ID does not exist" containerID="ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"
Sep 09 21:32:51.655908 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.655825    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68"} err="failed to get container status \"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68\": rpc error: code = NotFound desc = could not find container \"ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68\": container with ID starting with ff0cf311424245264662e094b6259b787d9f1288e4aabb24e4380ae30fb3ef68 not found: ID does not exist"
Sep 09 21:32:51.658514 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.658493    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:32:51.660433 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.660412    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-skk2b"]
Sep 09 21:32:51.668175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.668148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.668281 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.668196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.668281 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.668237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.668281 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.668264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfmz\" (UniqueName: \"kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.668399 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.668298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.768878 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.768841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769063 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.768891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769063 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:51.768988    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 09 21:32:51.769063 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfmz\" (UniqueName: \"kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769063 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769320 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:32:51.769070    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls podName:250e2f85-de45-42d6-be14-7652d30f4a65 nodeName:}" failed. No retries permitted until 2026-09-09 21:32:52.269047259 +0000 UTC m=+2451.765321293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 09 21:32:51.769320 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.769636 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.769616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:51.778769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:51.778750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfmz\" (UniqueName: \"kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:52.273761 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:52.273694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:52.276234 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:52.276197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:52.484846 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:52.484788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:52.761057 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:52.761032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:32:52.763351 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:32:52.763324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250e2f85_de45_42d6_be14_7652d30f4a65.slice/crio-ae18028420d87ef38763096bb6e48ca553fde394dd71462a6d665f87e6cf13b7 WatchSource:0}: Error finding container ae18028420d87ef38763096bb6e48ca553fde394dd71462a6d665f87e6cf13b7: Status 404 returned error can't find the container with id ae18028420d87ef38763096bb6e48ca553fde394dd71462a6d665f87e6cf13b7
Sep 09 21:32:53.029648 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:53.029611    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6731bde2-60ee-4839-8972-1656fa360ecb" path="/var/lib/kubelet/pods/6731bde2-60ee-4839-8972-1656fa360ecb/volumes"
Sep 09 21:32:53.642443 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:53.642411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerStarted","Data":"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"}
Sep 09 21:32:53.642615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:53.642464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerStarted","Data":"ae18028420d87ef38763096bb6e48ca553fde394dd71462a6d665f87e6cf13b7"}
Sep 09 21:32:57.657173 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:57.657136    2581 generic.go:356] "Generic (PLEG): container finished" podID="250e2f85-de45-42d6-be14-7652d30f4a65" containerID="9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef" exitCode=0
Sep 09 21:32:57.657615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:57.657209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerDied","Data":"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"}
Sep 09 21:32:58.663337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.663295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerStarted","Data":"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"}
Sep 09 21:32:58.663780 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.663363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerStarted","Data":"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"}
Sep 09 21:32:58.663780 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.663641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:58.663901 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.663818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:32:58.665015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.664995    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:32:58.688092 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:58.688037    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podStartSLOduration=3.754065582 podStartE2EDuration="7.68801926s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.933953678s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:32:58.686295154 +0000 UTC m=+2458.182569197" watchObservedRunningTime="2026-09-09 21:32:58.68801926 +0000 UTC m=+2458.184293301"
Sep 09 21:32:59.668222 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:32:59.668184    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:04.673042 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:04.673012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:33:04.675372 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:04.673491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:11.429690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.429653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:33:11.433040 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.433021    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.435905 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.435881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 09 21:33:11.436021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.435887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:33:11.445020 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.444992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:33:11.523052 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.523012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.523243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.523070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.523243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.523096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw9p\" (UniqueName: \"kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.523243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.523113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.523243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.523137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw9p\" (UniqueName: \"kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624668 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.624919 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.624894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.625144 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.625122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.625206 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.625174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.627207 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.627178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.633551 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.633526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw9p\" (UniqueName: \"kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p\") pod \"isvc-xgboost-v2-predictor-fcc668bc-kw5hr\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:11.743559 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:11.743463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:12.019448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:12.019421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:33:12.021199 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:33:12.021162    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d662259_a18f_49b3_85bf_7f6c8074ffba.slice/crio-a56f9174fbd9165b6cb75b44195d79ab9c5b723800acd3b47594fb82ca1fcec7 WatchSource:0}: Error finding container a56f9174fbd9165b6cb75b44195d79ab9c5b723800acd3b47594fb82ca1fcec7: Status 404 returned error can't find the container with id a56f9174fbd9165b6cb75b44195d79ab9c5b723800acd3b47594fb82ca1fcec7
Sep 09 21:33:12.720239 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:12.720203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerStarted","Data":"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"}
Sep 09 21:33:12.720619 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:12.720265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerStarted","Data":"a56f9174fbd9165b6cb75b44195d79ab9c5b723800acd3b47594fb82ca1fcec7"}
Sep 09 21:33:14.674018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:14.673979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:16.737856 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:16.737821    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerID="00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913" exitCode=0
Sep 09 21:33:16.738249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:16.737894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerDied","Data":"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"}
Sep 09 21:33:17.744444 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:17.744406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerStarted","Data":"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"}
Sep 09 21:33:17.744865 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:17.744473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerStarted","Data":"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"}
Sep 09 21:33:17.744865 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:17.744748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:17.770163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:17.770103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podStartSLOduration=2.6869038610000002 podStartE2EDuration="6.770085314s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.083181453s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:17.768694589 +0000 UTC m=+2477.264968664" watchObservedRunningTime="2026-09-09 21:33:17.770085314 +0000 UTC m=+2477.266359356"
Sep 09 21:33:18.748257 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:18.748222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:18.749969 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:18.749923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:33:19.751883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:19.751840    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:33:24.673752 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:24.673690    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:24.756282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:24.756254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:33:24.756782 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:24.756751    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:33:34.674196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:34.674157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:34.757095 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:34.757056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:33:44.674166 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:44.674124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:44.756879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:44.756841    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:33:54.674487 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:54.674395    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:33:54.757736 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:33:54.757678    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:34:04.673563 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:04.673520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:34:04.756944 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:04.756906    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:34:14.674999 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:14.674966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:34:14.757501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:14.757458    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:34:21.949272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:21.949225    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:34:21.949822 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:21.949474    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" containerID="cri-o://d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72" gracePeriod=30
Sep 09 21:34:21.949822 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:21.949564    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kube-rbac-proxy" containerID="cri-o://a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2" gracePeriod=30
Sep 09 21:34:22.998607 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:22.998575    2581 generic.go:356] "Generic (PLEG): container finished" podID="250e2f85-de45-42d6-be14-7652d30f4a65" containerID="a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2" exitCode=2
Sep 09 21:34:22.999016 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:22.998632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerDied","Data":"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"}
Sep 09 21:34:24.669138 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:24.669093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 09 21:34:24.674454 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:24.674427    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 09 21:34:24.757912 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:24.757879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:34:26.683446 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.683420    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:34:26.749068 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location\") pod \"250e2f85-de45-42d6-be14-7652d30f4a65\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") "
Sep 09 21:34:26.749265 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") pod \"250e2f85-de45-42d6-be14-7652d30f4a65\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") "
Sep 09 21:34:26.749265 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config\") pod \"250e2f85-de45-42d6-be14-7652d30f4a65\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") "
Sep 09 21:34:26.749265 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749116    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert\") pod \"250e2f85-de45-42d6-be14-7652d30f4a65\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") "
Sep 09 21:34:26.749265 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkfmz\" (UniqueName: \"kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz\") pod \"250e2f85-de45-42d6-be14-7652d30f4a65\" (UID: \"250e2f85-de45-42d6-be14-7652d30f4a65\") "
Sep 09 21:34:26.749490 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "250e2f85-de45-42d6-be14-7652d30f4a65" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:26.749534 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749479    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "250e2f85-de45-42d6-be14-7652d30f4a65" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:34:26.749569 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.749553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "250e2f85-de45-42d6-be14-7652d30f4a65" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:34:26.751223 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.751202    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "250e2f85-de45-42d6-be14-7652d30f4a65" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:34:26.751313 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.751263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz" (OuterVolumeSpecName: "kube-api-access-xkfmz") pod "250e2f85-de45-42d6-be14-7652d30f4a65" (UID: "250e2f85-de45-42d6-be14-7652d30f4a65"). InnerVolumeSpecName "kube-api-access-xkfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:34:26.849879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.849788    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/250e2f85-de45-42d6-be14-7652d30f4a65-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:26.849879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.849820    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/250e2f85-de45-42d6-be14-7652d30f4a65-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:26.849879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.849830    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:26.849879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.849839    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/250e2f85-de45-42d6-be14-7652d30f4a65-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:26.849879 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:26.849849    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkfmz\" (UniqueName: \"kubernetes.io/projected/250e2f85-de45-42d6-be14-7652d30f4a65-kube-api-access-xkfmz\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:27.015757 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.015724    2581 generic.go:356] "Generic (PLEG): container finished" podID="250e2f85-de45-42d6-be14-7652d30f4a65" containerID="d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72" exitCode=0
Sep 09 21:34:27.015920 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.015779    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"
Sep 09 21:34:27.015920 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.015797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerDied","Data":"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"}
Sep 09 21:34:27.015920 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.015855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n" event={"ID":"250e2f85-de45-42d6-be14-7652d30f4a65","Type":"ContainerDied","Data":"ae18028420d87ef38763096bb6e48ca553fde394dd71462a6d665f87e6cf13b7"}
Sep 09 21:34:27.015920 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.015881    2581 scope.go:117] "RemoveContainer" containerID="a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"
Sep 09 21:34:27.024567 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.024546    2581 scope.go:117] "RemoveContainer" containerID="d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"
Sep 09 21:34:27.032474 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.032455    2581 scope.go:117] "RemoveContainer" containerID="9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"
Sep 09 21:34:27.039874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.039846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:34:27.041057 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.041040    2581 scope.go:117] "RemoveContainer" containerID="a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"
Sep 09 21:34:27.041383 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:27.041354    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2\": container with ID starting with a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2 not found: ID does not exist" containerID="a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"
Sep 09 21:34:27.041466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.041393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2"} err="failed to get container status \"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2\": rpc error: code = NotFound desc = could not find container \"a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2\": container with ID starting with a7d02f908257df6d21a297a37980345910390c4eacb8cfde4ebfde3c1edd31b2 not found: ID does not exist"
Sep 09 21:34:27.041466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.041412    2581 scope.go:117] "RemoveContainer" containerID="d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"
Sep 09 21:34:27.041857 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:27.041830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72\": container with ID starting with d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72 not found: ID does not exist" containerID="d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"
Sep 09 21:34:27.041980 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.041867    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72"} err="failed to get container status \"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72\": rpc error: code = NotFound desc = could not find container \"d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72\": container with ID starting with d25ee20501cffaaacc497a8a3b04899beaf30896bd375c9469e1f9250e2f3c72 not found: ID does not exist"
Sep 09 21:34:27.041980 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.041890    2581 scope.go:117] "RemoveContainer" containerID="9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"
Sep 09 21:34:27.042349 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:27.042323    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef\": container with ID starting with 9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef not found: ID does not exist" containerID="9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"
Sep 09 21:34:27.042438 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.042354    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef"} err="failed to get container status \"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef\": rpc error: code = NotFound desc = could not find container \"9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef\": container with ID starting with 9206aa61be098001e63f1996ceb19dace89fb2f6aa7bc71b76ddb2a96b8f39ef not found: ID does not exist"
Sep 09 21:34:27.043268 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:27.043250    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-lvc2n"]
Sep 09 21:34:29.028836 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:29.028804    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" path="/var/lib/kubelet/pods/250e2f85-de45-42d6-be14-7652d30f4a65/volumes"
Sep 09 21:34:31.462099 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:31.462065    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:34:31.462508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:31.462310    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" containerID="cri-o://5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06" gracePeriod=30
Sep 09 21:34:31.462508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:31.462335    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kube-rbac-proxy" containerID="cri-o://1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f" gracePeriod=30
Sep 09 21:34:32.037539 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:32.037506    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerID="1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f" exitCode=2
Sep 09 21:34:32.037744 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:32.037582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerDied","Data":"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"}
Sep 09 21:34:34.752254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:34.752216    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 09 21:34:34.757600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:34.757571    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 21:34:35.309105 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.309077    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:34:35.414448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls\") pod \"1d662259-a18f-49b3-85bf-7f6c8074ffba\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") "
Sep 09 21:34:35.414640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert\") pod \"1d662259-a18f-49b3-85bf-7f6c8074ffba\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") "
Sep 09 21:34:35.414640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location\") pod \"1d662259-a18f-49b3-85bf-7f6c8074ffba\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") "
Sep 09 21:34:35.414640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config\") pod \"1d662259-a18f-49b3-85bf-7f6c8074ffba\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") "
Sep 09 21:34:35.414640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfw9p\" (UniqueName: \"kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p\") pod \"1d662259-a18f-49b3-85bf-7f6c8074ffba\" (UID: \"1d662259-a18f-49b3-85bf-7f6c8074ffba\") "
Sep 09 21:34:35.414958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d662259-a18f-49b3-85bf-7f6c8074ffba" (UID: "1d662259-a18f-49b3-85bf-7f6c8074ffba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:34:35.414958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d662259-a18f-49b3-85bf-7f6c8074ffba" (UID: "1d662259-a18f-49b3-85bf-7f6c8074ffba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:34:35.415077 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.414936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d662259-a18f-49b3-85bf-7f6c8074ffba" (UID: "1d662259-a18f-49b3-85bf-7f6c8074ffba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:34:35.416754 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.416730    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p" (OuterVolumeSpecName: "kube-api-access-zfw9p") pod "1d662259-a18f-49b3-85bf-7f6c8074ffba" (UID: "1d662259-a18f-49b3-85bf-7f6c8074ffba"). InnerVolumeSpecName "kube-api-access-zfw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:34:35.416855 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.416834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d662259-a18f-49b3-85bf-7f6c8074ffba" (UID: "1d662259-a18f-49b3-85bf-7f6c8074ffba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:34:35.515425 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.515371    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:35.515425 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.515423    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d662259-a18f-49b3-85bf-7f6c8074ffba-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:35.515623 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.515435    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:35.515623 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.515448    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfw9p\" (UniqueName: \"kubernetes.io/projected/1d662259-a18f-49b3-85bf-7f6c8074ffba-kube-api-access-zfw9p\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:35.515623 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:35.515461    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d662259-a18f-49b3-85bf-7f6c8074ffba-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:34:36.054150 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.054116    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerID="5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06" exitCode=0
Sep 09 21:34:36.054758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.054167    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"
Sep 09 21:34:36.054758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.054205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerDied","Data":"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"}
Sep 09 21:34:36.054758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.054281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr" event={"ID":"1d662259-a18f-49b3-85bf-7f6c8074ffba","Type":"ContainerDied","Data":"a56f9174fbd9165b6cb75b44195d79ab9c5b723800acd3b47594fb82ca1fcec7"}
Sep 09 21:34:36.054758 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.054322    2581 scope.go:117] "RemoveContainer" containerID="1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"
Sep 09 21:34:36.064276 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.064249    2581 scope.go:117] "RemoveContainer" containerID="5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"
Sep 09 21:34:36.072726 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.072677    2581 scope.go:117] "RemoveContainer" containerID="00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"
Sep 09 21:34:36.080555 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.080525    2581 scope.go:117] "RemoveContainer" containerID="1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"
Sep 09 21:34:36.080953 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:36.080917    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f\": container with ID starting with 1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f not found: ID does not exist" containerID="1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"
Sep 09 21:34:36.081019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.080957    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f"} err="failed to get container status \"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f\": rpc error: code = NotFound desc = could not find container \"1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f\": container with ID starting with 1019b95b038dcbf5006bc50f406e55fefdf07dd5c2c65b8048f03078c8dfa26f not found: ID does not exist"
Sep 09 21:34:36.081019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.080983    2581 scope.go:117] "RemoveContainer" containerID="5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"
Sep 09 21:34:36.081341 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:36.081316    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06\": container with ID starting with 5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06 not found: ID does not exist" containerID="5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"
Sep 09 21:34:36.081422 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.081348    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06"} err="failed to get container status \"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06\": rpc error: code = NotFound desc = could not find container \"5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06\": container with ID starting with 5786cf65079d5b46a9d38195d7fbf7ee4e1a838716980e2f31f8fc32e61ccd06 not found: ID does not exist"
Sep 09 21:34:36.081422 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.081378    2581 scope.go:117] "RemoveContainer" containerID="00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"
Sep 09 21:34:36.081618 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.081602    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:34:36.081691 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:34:36.081670    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913\": container with ID starting with 00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913 not found: ID does not exist" containerID="00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"
Sep 09 21:34:36.081755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.081718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913"} err="failed to get container status \"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913\": rpc error: code = NotFound desc = could not find container \"00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913\": container with ID starting with 00fc1d967620afeb50f8cbfd86d0a2ad9ce3c946a2964530d98b12b0a8259913 not found: ID does not exist"
Sep 09 21:34:36.084948 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.084928    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-kw5hr"]
Sep 09 21:34:36.667829 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.667795    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:34:36.668168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668150    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="storage-initializer"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="storage-initializer"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kube-rbac-proxy"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kube-rbac-proxy"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668215    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="storage-initializer"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668224    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="storage-initializer"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668244    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668254    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kube-rbac-proxy"
Sep 09 21:34:36.668262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668263    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kube-rbac-proxy"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668274    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668283    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668353    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kube-rbac-proxy"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668366    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kserve-container"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668374    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="250e2f85-de45-42d6-be14-7652d30f4a65" containerName="kserve-container"
Sep 09 21:34:36.668734 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.668383    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" containerName="kube-rbac-proxy"
Sep 09 21:34:36.672741 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.672717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.675648 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.675623    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:34:36.675811 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.675766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:34:36.676791 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.676773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:34:36.676902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.676788    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 09 21:34:36.676902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.676777    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 09 21:34:36.676902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.676859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:34:36.682358 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.682334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:34:36.726110 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.726074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7pq\" (UniqueName: \"kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.726110 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.726111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.726337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.726148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.726337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.726167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.826779 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.826742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.826779 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.826797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.827073 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.826874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7pq\" (UniqueName: \"kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.827073 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.826899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.827298 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.827273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.827603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.827580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.829432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.829413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.835388 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.835367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7pq\" (UniqueName: \"kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq\") pod \"isvc-sklearn-s3-predictor-6957ddcb79-brmk5\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:36.983608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:36.983517    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:37.029992 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:37.029959    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d662259-a18f-49b3-85bf-7f6c8074ffba" path="/var/lib/kubelet/pods/1d662259-a18f-49b3-85bf-7f6c8074ffba/volumes"
Sep 09 21:34:37.262440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:37.262358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:34:37.265439 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:34:37.265413    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7676df_d912_4f35_a796_a0cb5c3bc2d9.slice/crio-e435c65556ea97123bc1b5526e3f762712177a58a1142794deb1c0dc9279f46d WatchSource:0}: Error finding container e435c65556ea97123bc1b5526e3f762712177a58a1142794deb1c0dc9279f46d: Status 404 returned error can't find the container with id e435c65556ea97123bc1b5526e3f762712177a58a1142794deb1c0dc9279f46d
Sep 09 21:34:38.065581 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:38.065547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerStarted","Data":"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"}
Sep 09 21:34:38.065765 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:38.065607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerStarted","Data":"e435c65556ea97123bc1b5526e3f762712177a58a1142794deb1c0dc9279f46d"}
Sep 09 21:34:39.070790 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:39.070758    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerID="efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7" exitCode=0
Sep 09 21:34:39.071169 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:39.070823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerDied","Data":"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"}
Sep 09 21:34:40.076902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:40.076860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerStarted","Data":"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"}
Sep 09 21:34:40.077290 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:40.076936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerStarted","Data":"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"}
Sep 09 21:34:40.077290 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:40.077010    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:40.099678 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:40.099627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podStartSLOduration=2.979770347 podStartE2EDuration="4.099609265s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.119838918s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:40.098272788 +0000 UTC m=+2559.594546847" watchObservedRunningTime="2026-09-09 21:34:40.099609265 +0000 UTC m=+2559.595883306"
Sep 09 21:34:41.081031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:41.080990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:41.082249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:41.082223    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:34:42.084804 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:42.084764    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:34:47.089085 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:47.089056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:34:47.089572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:47.089542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:34:57.090267 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:34:57.090231    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:07.089862 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:07.089735    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:17.089556 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:17.089514    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:22.426904 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.426859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:22.430499 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.430474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.433386 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.433358    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:35:22.433386 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.433384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 09 21:35:22.433601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.433367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:35:22.433601 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.433475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:35:22.442612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.442585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:22.580781 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.580742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.580962 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.580801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4nb\" (UniqueName: \"kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.580962 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.580883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.580962 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.580922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.580962 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.580945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.681921 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.681831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.681921 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.681882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.681921 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.681912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.682171 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.681965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.682171 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.682131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4nb\" (UniqueName: \"kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.682331 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.682307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.682746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.682725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.682746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.682736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.684430 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.684412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.690887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.690861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4nb\" (UniqueName: \"kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:22.741009 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:22.740973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:23.015934 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:23.015904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:23.018437 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:35:23.018411    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cacacec_92b9_4ab2_93c2_07d3f56fc520.slice/crio-02bd3e528ebad1dc7d30e14f0e4c8297fc082940db0e2b99cf8b09d653ac9729 WatchSource:0}: Error finding container 02bd3e528ebad1dc7d30e14f0e4c8297fc082940db0e2b99cf8b09d653ac9729: Status 404 returned error can't find the container with id 02bd3e528ebad1dc7d30e14f0e4c8297fc082940db0e2b99cf8b09d653ac9729
Sep 09 21:35:23.245530 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:23.245486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerStarted","Data":"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"}
Sep 09 21:35:23.245719 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:23.245553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerStarted","Data":"02bd3e528ebad1dc7d30e14f0e4c8297fc082940db0e2b99cf8b09d653ac9729"}
Sep 09 21:35:27.090446 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:27.090405    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:27.263363 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:27.263331    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerID="f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92" exitCode=0
Sep 09 21:35:27.263539 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:27.263365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerDied","Data":"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"}
Sep 09 21:35:28.271323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:28.271284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerStarted","Data":"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"}
Sep 09 21:35:28.271797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:28.271348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerStarted","Data":"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"}
Sep 09 21:35:28.271797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:28.271634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:28.271797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:28.271792    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:28.273128 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:28.273103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 21:35:29.275662 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:29.275618    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 21:35:34.280996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:34.280967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:34.281488 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:34.281460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 21:35:34.305981 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:34.305912    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podStartSLOduration=8.362551217 podStartE2EDuration="12.305896657s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.94334544s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:28.306099989 +0000 UTC m=+2607.802374055" watchObservedRunningTime="2026-09-09 21:35:34.305896657 +0000 UTC m=+2613.802170699"
Sep 09 21:35:37.089569 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:37.089529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:44.282987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:44.282959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:47.089737 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:47.089672    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:52.474867 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:52.472756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:52.474867 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:52.473149    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container" containerID="cri-o://d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53" gracePeriod=30
Sep 09 21:35:52.474867 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:52.473459    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kube-rbac-proxy" containerID="cri-o://c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e" gracePeriod=30
Sep 09 21:35:53.114949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.114922    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:53.126261 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert\") pod \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") "
Sep 09 21:35:53.126261 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s4nb\" (UniqueName: \"kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb\") pod \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") "
Sep 09 21:35:53.126509 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location\") pod \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") "
Sep 09 21:35:53.126509 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config\") pod \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") "
Sep 09 21:35:53.126509 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls\") pod \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\" (UID: \"3cacacec-92b9-4ab2-93c2-07d3f56fc520\") "
Sep 09 21:35:53.126693 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cacacec-92b9-4ab2-93c2-07d3f56fc520" (UID: "3cacacec-92b9-4ab2-93c2-07d3f56fc520"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:53.126780 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cacacec-92b9-4ab2-93c2-07d3f56fc520" (UID: "3cacacec-92b9-4ab2-93c2-07d3f56fc520"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:53.126780 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.126683    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cacacec-92b9-4ab2-93c2-07d3f56fc520" (UID: "3cacacec-92b9-4ab2-93c2-07d3f56fc520"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:35:53.128504 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.128472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cacacec-92b9-4ab2-93c2-07d3f56fc520" (UID: "3cacacec-92b9-4ab2-93c2-07d3f56fc520"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:53.128642 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.128521    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb" (OuterVolumeSpecName: "kube-api-access-6s4nb") pod "3cacacec-92b9-4ab2-93c2-07d3f56fc520" (UID: "3cacacec-92b9-4ab2-93c2-07d3f56fc520"). InnerVolumeSpecName "kube-api-access-6s4nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:53.227323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.227225    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cacacec-92b9-4ab2-93c2-07d3f56fc520-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:35:53.227323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.227272    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:35:53.227323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.227289    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4nb\" (UniqueName: \"kubernetes.io/projected/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-api-access-6s4nb\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:35:53.227323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.227305    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:35:53.227323 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.227319    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cacacec-92b9-4ab2-93c2-07d3f56fc520-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:35:53.365339 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365307    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerID="c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e" exitCode=2
Sep 09 21:35:53.365339 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365331    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerID="d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53" exitCode=0
Sep 09 21:35:53.365592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"
Sep 09 21:35:53.365592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerDied","Data":"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"}
Sep 09 21:35:53.365592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerDied","Data":"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"}
Sep 09 21:35:53.365592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg" event={"ID":"3cacacec-92b9-4ab2-93c2-07d3f56fc520","Type":"ContainerDied","Data":"02bd3e528ebad1dc7d30e14f0e4c8297fc082940db0e2b99cf8b09d653ac9729"}
Sep 09 21:35:53.365592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.365538    2581 scope.go:117] "RemoveContainer" containerID="c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"
Sep 09 21:35:53.374898 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.374870    2581 scope.go:117] "RemoveContainer" containerID="d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"
Sep 09 21:35:53.382599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.382578    2581 scope.go:117] "RemoveContainer" containerID="f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"
Sep 09 21:35:53.391134 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.391092    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:53.394837 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.394626    2581 scope.go:117] "RemoveContainer" containerID="c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"
Sep 09 21:35:53.395196 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:35:53.395171    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e\": container with ID starting with c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e not found: ID does not exist" containerID="c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"
Sep 09 21:35:53.395557 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.395206    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"} err="failed to get container status \"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e\": rpc error: code = NotFound desc = could not find container \"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e\": container with ID starting with c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e not found: ID does not exist"
Sep 09 21:35:53.395557 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.395234    2581 scope.go:117] "RemoveContainer" containerID="d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"
Sep 09 21:35:53.395722 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:35:53.395554    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53\": container with ID starting with d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53 not found: ID does not exist" containerID="d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"
Sep 09 21:35:53.395722 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.395585    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"} err="failed to get container status \"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53\": rpc error: code = NotFound desc = could not find container \"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53\": container with ID starting with d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53 not found: ID does not exist"
Sep 09 21:35:53.395722 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.395608    2581 scope.go:117] "RemoveContainer" containerID="f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"
Sep 09 21:35:53.396047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396027    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-62clg"]
Sep 09 21:35:53.396359 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:35:53.396335    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92\": container with ID starting with f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92 not found: ID does not exist" containerID="f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"
Sep 09 21:35:53.396452 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396364    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"} err="failed to get container status \"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92\": rpc error: code = NotFound desc = could not find container \"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92\": container with ID starting with f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92 not found: ID does not exist"
Sep 09 21:35:53.396452 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396380    2581 scope.go:117] "RemoveContainer" containerID="c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"
Sep 09 21:35:53.396646 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396625    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e"} err="failed to get container status \"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e\": rpc error: code = NotFound desc = could not find container \"c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e\": container with ID starting with c7721fa502b2c74d8228e9a38652b298b9b62ebe2f6f8e69c3f797747f6ab59e not found: ID does not exist"
Sep 09 21:35:53.396738 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396648    2581 scope.go:117] "RemoveContainer" containerID="d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"
Sep 09 21:35:53.396967 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396943    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53"} err="failed to get container status \"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53\": rpc error: code = NotFound desc = could not find container \"d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53\": container with ID starting with d68b1d6e79e44f8a2a9de1f8f8d0f21aa043341070c230bac471611bffe9bf53 not found: ID does not exist"
Sep 09 21:35:53.397021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.396970    2581 scope.go:117] "RemoveContainer" containerID="f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"
Sep 09 21:35:53.397237 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:53.397218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92"} err="failed to get container status \"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92\": rpc error: code = NotFound desc = could not find container \"f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92\": container with ID starting with f6d91e40befbeba0dc055c440e615ba3cb3469225c79726e396db5889e497a92 not found: ID does not exist"
Sep 09 21:35:54.024864 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:54.024822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:35:55.029653 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:35:55.029623    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" path="/var/lib/kubelet/pods/3cacacec-92b9-4ab2-93c2-07d3f56fc520/volumes"
Sep 09 21:36:04.025620 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:04.025585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:36:10.843023 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:10.842988    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:36:10.843431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:10.843294    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" containerID="cri-o://515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8" gracePeriod=30
Sep 09 21:36:10.843431 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:10.843367    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kube-rbac-proxy" containerID="cri-o://4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d" gracePeriod=30
Sep 09 21:36:11.002622 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002585    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:36:11.002881 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002869    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="storage-initializer"
Sep 09 21:36:11.002929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002882    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="storage-initializer"
Sep 09 21:36:11.002929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002899    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kube-rbac-proxy"
Sep 09 21:36:11.002929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002905    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kube-rbac-proxy"
Sep 09 21:36:11.002929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002913    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container"
Sep 09 21:36:11.002929 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002919    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container"
Sep 09 21:36:11.003118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002962    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kube-rbac-proxy"
Sep 09 21:36:11.003118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.002971    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cacacec-92b9-4ab2-93c2-07d3f56fc520" containerName="kserve-container"
Sep 09 21:36:11.005944 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.005925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.008573 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.008547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 09 21:36:11.008846 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.008823    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 09 21:36:11.008964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.008877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:36:11.018004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.017976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:36:11.062603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.062568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.062603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.062606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9rn\" (UniqueName: \"kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.062839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.062633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.062839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.062751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.062839 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.062834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.163670 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.163632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.163670 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.163676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.163945 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.163720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9rn\" (UniqueName: \"kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.163945 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.163753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.163945 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.163782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.164229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.164207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.164437 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.164414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.164521 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.164499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.166204 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.166186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.172346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.172320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9rn\" (UniqueName: \"kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.318676 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.318636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:11.434836 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.434734    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerID="4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d" exitCode=2
Sep 09 21:36:11.434836 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.434804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerDied","Data":"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"}
Sep 09 21:36:11.592009 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:11.591976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:36:11.594903 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:36:11.594877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172df869_645e_4ede_966c_0d6f5aedf0ce.slice/crio-221733aa5a1bbc281c3d07c23142f9c41d9f9c4e20907423cde4152400749231 WatchSource:0}: Error finding container 221733aa5a1bbc281c3d07c23142f9c41d9f9c4e20907423cde4152400749231: Status 404 returned error can't find the container with id 221733aa5a1bbc281c3d07c23142f9c41d9f9c4e20907423cde4152400749231
Sep 09 21:36:12.085829 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:12.085785    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 09 21:36:12.440477 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:12.440439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerStarted","Data":"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"}
Sep 09 21:36:12.440653 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:12.440525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerStarted","Data":"221733aa5a1bbc281c3d07c23142f9c41d9f9c4e20907423cde4152400749231"}
Sep 09 21:36:13.447043 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:13.447001    2581 generic.go:356] "Generic (PLEG): container finished" podID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerID="909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca" exitCode=0
Sep 09 21:36:13.447526 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:13.447077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerDied","Data":"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"}
Sep 09 21:36:14.024860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.024785    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 21:36:14.452592 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.452554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerStarted","Data":"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"}
Sep 09 21:36:14.453175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.452609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerStarted","Data":"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"}
Sep 09 21:36:14.453175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.452722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:14.453175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.452864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:14.454186 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.454162    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:14.475691 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:14.475644    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podStartSLOduration=3.394979976 podStartE2EDuration="4.475629567s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.080649591s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:14.47474142 +0000 UTC m=+2653.971015476" watchObservedRunningTime="2026-09-09 21:36:14.475629567 +0000 UTC m=+2653.971903609"
Sep 09 21:36:15.456913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.456873    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:15.695502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.695476    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:36:15.802600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location\") pod \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") "
Sep 09 21:36:15.802600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g7pq\" (UniqueName: \"kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq\") pod \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") "
Sep 09 21:36:15.802600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config\") pod \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") "
Sep 09 21:36:15.802915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802615    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls\") pod \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\" (UID: \"ab7676df-d912-4f35-a796-a0cb5c3bc2d9\") "
Sep 09 21:36:15.802915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab7676df-d912-4f35-a796-a0cb5c3bc2d9" (UID: "ab7676df-d912-4f35-a796-a0cb5c3bc2d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:15.803046 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.802957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab7676df-d912-4f35-a796-a0cb5c3bc2d9" (UID: "ab7676df-d912-4f35-a796-a0cb5c3bc2d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:15.804583 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.804561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq" (OuterVolumeSpecName: "kube-api-access-5g7pq") pod "ab7676df-d912-4f35-a796-a0cb5c3bc2d9" (UID: "ab7676df-d912-4f35-a796-a0cb5c3bc2d9"). InnerVolumeSpecName "kube-api-access-5g7pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:36:15.804686 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.804661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab7676df-d912-4f35-a796-a0cb5c3bc2d9" (UID: "ab7676df-d912-4f35-a796-a0cb5c3bc2d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:15.903272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.903234    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:36:15.903272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.903264    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7pq\" (UniqueName: \"kubernetes.io/projected/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-api-access-5g7pq\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:36:15.903272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.903275    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:36:15.903510 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:15.903286    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7676df-d912-4f35-a796-a0cb5c3bc2d9-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:36:16.462596 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.462562    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerID="515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8" exitCode=0
Sep 09 21:36:16.463000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.462619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"
Sep 09 21:36:16.463000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.462640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerDied","Data":"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"}
Sep 09 21:36:16.463000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.462768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5" event={"ID":"ab7676df-d912-4f35-a796-a0cb5c3bc2d9","Type":"ContainerDied","Data":"e435c65556ea97123bc1b5526e3f762712177a58a1142794deb1c0dc9279f46d"}
Sep 09 21:36:16.463000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.462790    2581 scope.go:117] "RemoveContainer" containerID="4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"
Sep 09 21:36:16.471093 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.471078    2581 scope.go:117] "RemoveContainer" containerID="515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"
Sep 09 21:36:16.478540 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.478521    2581 scope.go:117] "RemoveContainer" containerID="efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"
Sep 09 21:36:16.486133 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.486112    2581 scope.go:117] "RemoveContainer" containerID="4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"
Sep 09 21:36:16.486205 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.486129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:36:16.486466 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:36:16.486448    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d\": container with ID starting with 4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d not found: ID does not exist" containerID="4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"
Sep 09 21:36:16.486516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.486476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d"} err="failed to get container status \"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d\": rpc error: code = NotFound desc = could not find container \"4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d\": container with ID starting with 4254ce10c5b39da64c41f6186e5c03e765895c74e3d359997dea6484006fc30d not found: ID does not exist"
Sep 09 21:36:16.486516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.486494    2581 scope.go:117] "RemoveContainer" containerID="515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"
Sep 09 21:36:16.489659 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:36:16.489330    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8\": container with ID starting with 515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8 not found: ID does not exist" containerID="515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"
Sep 09 21:36:16.489659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.489388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8"} err="failed to get container status \"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8\": rpc error: code = NotFound desc = could not find container \"515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8\": container with ID starting with 515af8ce082a301f3acfef0103b94f568396b2540d129f0ef8d9c455e0a002c8 not found: ID does not exist"
Sep 09 21:36:16.489659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.489413    2581 scope.go:117] "RemoveContainer" containerID="efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"
Sep 09 21:36:16.490252 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:36:16.490224    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7\": container with ID starting with efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7 not found: ID does not exist" containerID="efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"
Sep 09 21:36:16.490350 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.490260    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7"} err="failed to get container status \"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7\": rpc error: code = NotFound desc = could not find container \"efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7\": container with ID starting with efaf58a7dc65752c72aa51381c6cb56414eff28a48916f25c6752edf777f10a7 not found: ID does not exist"
Sep 09 21:36:16.491199 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:16.491181    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6957ddcb79-brmk5"]
Sep 09 21:36:17.029236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:17.029207    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" path="/var/lib/kubelet/pods/ab7676df-d912-4f35-a796-a0cb5c3bc2d9/volumes"
Sep 09 21:36:20.461590 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:20.461557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:36:20.462069 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:20.462042    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:30.462860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:30.462822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:40.462787 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:40.462744    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:50.462130 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:50.462087    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:36:52.863243 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863205    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863507    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kube-rbac-proxy"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863518    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kube-rbac-proxy"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863530    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863536    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863547    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="storage-initializer"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863553    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="storage-initializer"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863605    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kube-rbac-proxy"
Sep 09 21:36:52.863608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.863613    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7676df-d912-4f35-a796-a0cb5c3bc2d9" containerName="kserve-container"
Sep 09 21:36:52.867032 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.867010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:52.869667 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.869637    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 09 21:36:52.869667 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.869660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 09 21:36:52.869973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.869730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:36:52.879203 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:52.879177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:36:53.001118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.001080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.001346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.001127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.001346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.001151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d4p\" (UniqueName: \"kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.001346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.001243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.001346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.001292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102392 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102392 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102392 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d4p\" (UniqueName: \"kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.102690 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:36:53.102543    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 09 21:36:53.102690 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:36:53.102632    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls podName:0242520b-6e9e-485f-b3f0-a4278e0403fc nodeName:}" failed. No retries permitted until 2026-09-09 21:36:53.602608618 +0000 UTC m=+2693.098882640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls") pod "isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 09 21:36:53.102958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.102825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.103172 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.103152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.103244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.103184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.111167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.111145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d4p\" (UniqueName: \"kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.607683 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.607639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.610130 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.610106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:53.777790 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:53.777752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:36:54.054653 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:54.054630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:36:54.056834 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:36:54.056807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0242520b_6e9e_485f_b3f0_a4278e0403fc.slice/crio-3c78f839bf28128db304cf970dc06aadcdb8e2b3c24381845d255b75752dd21c WatchSource:0}: Error finding container 3c78f839bf28128db304cf970dc06aadcdb8e2b3c24381845d255b75752dd21c: Status 404 returned error can't find the container with id 3c78f839bf28128db304cf970dc06aadcdb8e2b3c24381845d255b75752dd21c
Sep 09 21:36:54.604230 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:54.604191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerStarted","Data":"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"}
Sep 09 21:36:54.604407 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:54.604268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerStarted","Data":"3c78f839bf28128db304cf970dc06aadcdb8e2b3c24381845d255b75752dd21c"}
Sep 09 21:36:58.620640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:58.620551    2581 generic.go:356] "Generic (PLEG): container finished" podID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerID="5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf" exitCode=0
Sep 09 21:36:58.620640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:58.620598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerDied","Data":"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"}
Sep 09 21:36:58.683013 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:58.682985    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:36:59.629277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:36:59.629193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerStarted","Data":"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"}
Sep 09 21:37:00.462772 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:00.462024    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:37:01.547282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:01.547252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:37:01.547730 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:01.547252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:37:02.645368 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.645333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerStarted","Data":"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"}
Sep 09 21:37:02.645753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.645399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerStarted","Data":"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"}
Sep 09 21:37:02.645753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.645509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:37:02.645753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.645572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:37:02.645753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.645598    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:37:02.666417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:02.666366    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podStartSLOduration=3.72290345 podStartE2EDuration="10.666348851s" totalImagesPullingTime="2.906871618s" totalInitContainerRuntime="4.036573783s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:37:02.66436102 +0000 UTC m=+2702.160635082" watchObservedRunningTime="2026-09-09 21:37:02.666348851 +0000 UTC m=+2702.162622893"
Sep 09 21:37:08.654434 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:08.654403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:37:10.462103 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:10.462063    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:37:20.462580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:20.462533    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:37:25.025259 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:25.025212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:37:28.655806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:28.655767    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 21:37:33.778292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:33.778242    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 21:37:35.030346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:35.030308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:37:38.658979 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:38.658947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:37:40.951428 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:40.951391    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:37:40.951862 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:40.951679    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" containerID="cri-o://36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550" gracePeriod=30
Sep 09 21:37:40.951862 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:40.951737    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kube-rbac-proxy" containerID="cri-o://2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9" gracePeriod=30
Sep 09 21:37:41.798778 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:41.798744    2581 generic.go:356] "Generic (PLEG): container finished" podID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerID="2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9" exitCode=2
Sep 09 21:37:41.798954 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:41.798817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerDied","Data":"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"}
Sep 09 21:37:45.025759 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.025716    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 21:37:45.457875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.457830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 09 21:37:45.801529 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.801504    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:37:45.815162 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.815129    2581 generic.go:356] "Generic (PLEG): container finished" podID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerID="36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550" exitCode=0
Sep 09 21:37:45.815302 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.815182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerDied","Data":"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"}
Sep 09 21:37:45.815302 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.815230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2" event={"ID":"172df869-645e-4ede-966c-0d6f5aedf0ce","Type":"ContainerDied","Data":"221733aa5a1bbc281c3d07c23142f9c41d9f9c4e20907423cde4152400749231"}
Sep 09 21:37:45.815302 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.815184    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"
Sep 09 21:37:45.815407 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.815258    2581 scope.go:117] "RemoveContainer" containerID="2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"
Sep 09 21:37:45.823147 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.823127    2581 scope.go:117] "RemoveContainer" containerID="36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"
Sep 09 21:37:45.831310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.831284    2581 scope.go:117] "RemoveContainer" containerID="909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"
Sep 09 21:37:45.840890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.840868    2581 scope.go:117] "RemoveContainer" containerID="2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"
Sep 09 21:37:45.841247 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:37:45.841224    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9\": container with ID starting with 2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9 not found: ID does not exist" containerID="2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"
Sep 09 21:37:45.841327 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.841257    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9"} err="failed to get container status \"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9\": rpc error: code = NotFound desc = could not find container \"2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9\": container with ID starting with 2874050cba1c0f15d6e3afac0591822eb5396e2a85db9f757b101a85659104a9 not found: ID does not exist"
Sep 09 21:37:45.841327 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.841275    2581 scope.go:117] "RemoveContainer" containerID="36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"
Sep 09 21:37:45.841618 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:37:45.841599    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550\": container with ID starting with 36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550 not found: ID does not exist" containerID="36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"
Sep 09 21:37:45.841683 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.841628    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550"} err="failed to get container status \"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550\": rpc error: code = NotFound desc = could not find container \"36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550\": container with ID starting with 36af4c542842995e89641411e7ede47d08ef760122e9f7a9a357807a7918a550 not found: ID does not exist"
Sep 09 21:37:45.841683 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.841651    2581 scope.go:117] "RemoveContainer" containerID="909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"
Sep 09 21:37:45.841996 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:37:45.841979    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca\": container with ID starting with 909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca not found: ID does not exist" containerID="909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"
Sep 09 21:37:45.842052 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.842001    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca"} err="failed to get container status \"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca\": rpc error: code = NotFound desc = could not find container \"909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca\": container with ID starting with 909347f8cc62b1459a905bd687a6f1eef709315856a45d7cc5e1b31659291bca not found: ID does not exist"
Sep 09 21:37:45.899262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls\") pod \"172df869-645e-4ede-966c-0d6f5aedf0ce\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") "
Sep 09 21:37:45.899262 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location\") pod \"172df869-645e-4ede-966c-0d6f5aedf0ce\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") "
Sep 09 21:37:45.899523 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv9rn\" (UniqueName: \"kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn\") pod \"172df869-645e-4ede-966c-0d6f5aedf0ce\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") "
Sep 09 21:37:45.899523 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config\") pod \"172df869-645e-4ede-966c-0d6f5aedf0ce\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") "
Sep 09 21:37:45.899523 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert\") pod \"172df869-645e-4ede-966c-0d6f5aedf0ce\" (UID: \"172df869-645e-4ede-966c-0d6f5aedf0ce\") "
Sep 09 21:37:45.899760 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "172df869-645e-4ede-966c-0d6f5aedf0ce" (UID: "172df869-645e-4ede-966c-0d6f5aedf0ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:45.899760 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "172df869-645e-4ede-966c-0d6f5aedf0ce" (UID: "172df869-645e-4ede-966c-0d6f5aedf0ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:45.899907 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.899776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "172df869-645e-4ede-966c-0d6f5aedf0ce" (UID: "172df869-645e-4ede-966c-0d6f5aedf0ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:45.901441 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.901423    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "172df869-645e-4ede-966c-0d6f5aedf0ce" (UID: "172df869-645e-4ede-966c-0d6f5aedf0ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:45.901582 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:45.901562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn" (OuterVolumeSpecName: "kube-api-access-gv9rn") pod "172df869-645e-4ede-966c-0d6f5aedf0ce" (UID: "172df869-645e-4ede-966c-0d6f5aedf0ce"). InnerVolumeSpecName "kube-api-access-gv9rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:46.000334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.000246    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9rn\" (UniqueName: \"kubernetes.io/projected/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-api-access-gv9rn\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:37:46.000334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.000274    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:37:46.000334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.000284    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/172df869-645e-4ede-966c-0d6f5aedf0ce-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:37:46.000334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.000293    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/172df869-645e-4ede-966c-0d6f5aedf0ce-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:37:46.000334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.000303    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/172df869-645e-4ede-966c-0d6f5aedf0ce-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:37:46.138198 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.138154    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:37:46.141380 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:46.141352    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-d67455747-9thh2"]
Sep 09 21:37:47.029594 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:37:47.029552    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" path="/var/lib/kubelet/pods/172df869-645e-4ede-966c-0d6f5aedf0ce/volumes"
Sep 09 21:38:08.660679 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:08.660645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:38:12.905685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:12.905648    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:38:12.906118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:12.905990    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" containerID="cri-o://ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445" gracePeriod=30
Sep 09 21:38:12.906118 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:12.906030    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-agent" containerID="cri-o://589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f" gracePeriod=30
Sep 09 21:38:12.906250 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:12.906001    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" containerID="cri-o://66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5" gracePeriod=30
Sep 09 21:38:13.649747 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:13.649677    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:13.924730 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:13.924626    2581 generic.go:356] "Generic (PLEG): container finished" podID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerID="66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5" exitCode=2
Sep 09 21:38:13.924730 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:13.924692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerDied","Data":"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"}
Sep 09 21:38:15.937820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:15.937694    2581 generic.go:356] "Generic (PLEG): container finished" podID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerID="ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445" exitCode=0
Sep 09 21:38:15.937820 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:15.937724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerDied","Data":"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"}
Sep 09 21:38:18.092563 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092528    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="storage-initializer"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092820    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="storage-initializer"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092834    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kube-rbac-proxy"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kube-rbac-proxy"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092855    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092904    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kube-rbac-proxy"
Sep 09 21:38:18.092985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.092910    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="172df869-645e-4ede-966c-0d6f5aedf0ce" containerName="kserve-container"
Sep 09 21:38:18.095947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.095929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.098686 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.098655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 09 21:38:18.098830 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.098746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 09 21:38:18.109661 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.109634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:38:18.159197 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.159156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.159376 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.159211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.159376 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.159304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.159376 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.159353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.159503 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.159405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260465 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260465 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260797 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.260990 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.260901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.261259 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.261232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.261259 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.261249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.263184 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.263166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.269891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.269866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmm89\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.406264 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.406222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:18.649799 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.649754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:18.655194 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.655161    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 21:38:18.676923 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.676831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:38:18.679842 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:38:18.679801    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f92c9cf_1064_4b64_8511_ea2f1a45da58.slice/crio-b6ab7e81bbb2d91af382429715195c6c40bdc22de35b0ede2432d9d422f7e89b WatchSource:0}: Error finding container b6ab7e81bbb2d91af382429715195c6c40bdc22de35b0ede2432d9d422f7e89b: Status 404 returned error can't find the container with id b6ab7e81bbb2d91af382429715195c6c40bdc22de35b0ede2432d9d422f7e89b
Sep 09 21:38:18.952588 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.952505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerStarted","Data":"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"}
Sep 09 21:38:18.952588 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:18.952563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerStarted","Data":"b6ab7e81bbb2d91af382429715195c6c40bdc22de35b0ede2432d9d422f7e89b"}
Sep 09 21:38:23.650392 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:23.650353    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:23.650853 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:23.650478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:38:23.974319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:23.974238    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerID="5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f" exitCode=0
Sep 09 21:38:23.974319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:23.974308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerDied","Data":"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"}
Sep 09 21:38:28.649908 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:28.649866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:28.655300 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:28.655266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 21:38:33.649657 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:33.649597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:36.031307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.031216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerStarted","Data":"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"}
Sep 09 21:38:36.031307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.031299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerStarted","Data":"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"}
Sep 09 21:38:36.031853 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.031620    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:36.031853 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.031691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:36.032963 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.032937    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:38:36.054926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:36.054871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podStartSLOduration=1.9585071520000001 podStartE2EDuration="18.054854386s" totalImagesPullingTime="11.690339432s" totalInitContainerRuntime="4.406007802s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:36.053343596 +0000 UTC m=+2795.549617634" watchObservedRunningTime="2026-09-09 21:38:36.054854386 +0000 UTC m=+2795.551128428"
Sep 09 21:38:37.036045 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:37.036008    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:38:38.650154 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:38.650102    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 21:38:38.655599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:38.655569    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 21:38:38.655753 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:38.655739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:38:42.040069 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:42.040039    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:38:42.040581 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:42.040553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:38:43.061155 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.061128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:38:43.061517 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.061424    2581 generic.go:356] "Generic (PLEG): container finished" podID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerID="589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f" exitCode=0
Sep 09 21:38:43.061517 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.061490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerDied","Data":"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"}
Sep 09 21:38:43.061596 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.061554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6" event={"ID":"0242520b-6e9e-485f-b3f0-a4278e0403fc","Type":"ContainerDied","Data":"3c78f839bf28128db304cf970dc06aadcdb8e2b3c24381845d255b75752dd21c"}
Sep 09 21:38:43.061596 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.061584    2581 scope.go:117] "RemoveContainer" containerID="66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"
Sep 09 21:38:43.069299 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.069284    2581 scope.go:117] "RemoveContainer" containerID="589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"
Sep 09 21:38:43.076579 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.076560    2581 scope.go:117] "RemoveContainer" containerID="ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"
Sep 09 21:38:43.083740 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.083721    2581 scope.go:117] "RemoveContainer" containerID="5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"
Sep 09 21:38:43.092410 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.092392    2581 scope.go:117] "RemoveContainer" containerID="66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"
Sep 09 21:38:43.092709 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:38:43.092682    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5\": container with ID starting with 66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5 not found: ID does not exist" containerID="66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"
Sep 09 21:38:43.092771 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.092731    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5"} err="failed to get container status \"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5\": rpc error: code = NotFound desc = could not find container \"66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5\": container with ID starting with 66c0d88a61ab444b959e02753e9330211f7271d88a4e60ce66f1145af6323bb5 not found: ID does not exist"
Sep 09 21:38:43.092771 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.092750    2581 scope.go:117] "RemoveContainer" containerID="589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"
Sep 09 21:38:43.093058 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:38:43.093033    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f\": container with ID starting with 589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f not found: ID does not exist" containerID="589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"
Sep 09 21:38:43.093112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.093070    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f"} err="failed to get container status \"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f\": rpc error: code = NotFound desc = could not find container \"589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f\": container with ID starting with 589c8751899b4848d9ab6a66d5a018369cb10af2be018a58142117f7ef8f233f not found: ID does not exist"
Sep 09 21:38:43.093112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.093098    2581 scope.go:117] "RemoveContainer" containerID="ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"
Sep 09 21:38:43.093379 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:38:43.093361    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445\": container with ID starting with ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445 not found: ID does not exist" containerID="ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"
Sep 09 21:38:43.093432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.093384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445"} err="failed to get container status \"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445\": rpc error: code = NotFound desc = could not find container \"ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445\": container with ID starting with ebaecef4598d592b528d561445f84baca5cf955ed76b53752efcfeeed93dc445 not found: ID does not exist"
Sep 09 21:38:43.093432 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.093400    2581 scope.go:117] "RemoveContainer" containerID="5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"
Sep 09 21:38:43.093643 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:38:43.093625    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf\": container with ID starting with 5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf not found: ID does not exist" containerID="5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"
Sep 09 21:38:43.093739 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.093651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf"} err="failed to get container status \"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf\": rpc error: code = NotFound desc = could not find container \"5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf\": container with ID starting with 5033c273a3d217edeb34ab31f70a14da8a656605fdddfd0d270355d6058982cf not found: ID does not exist"
Sep 09 21:38:43.169926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.169842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location\") pod \"0242520b-6e9e-485f-b3f0-a4278e0403fc\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") "
Sep 09 21:38:43.169926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.169889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config\") pod \"0242520b-6e9e-485f-b3f0-a4278e0403fc\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") "
Sep 09 21:38:43.169926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.169908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94d4p\" (UniqueName: \"kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p\") pod \"0242520b-6e9e-485f-b3f0-a4278e0403fc\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") "
Sep 09 21:38:43.170220 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.169941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert\") pod \"0242520b-6e9e-485f-b3f0-a4278e0403fc\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") "
Sep 09 21:38:43.170220 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.169980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") pod \"0242520b-6e9e-485f-b3f0-a4278e0403fc\" (UID: \"0242520b-6e9e-485f-b3f0-a4278e0403fc\") "
Sep 09 21:38:43.170335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.170240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0242520b-6e9e-485f-b3f0-a4278e0403fc" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:38:43.170335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.170238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0242520b-6e9e-485f-b3f0-a4278e0403fc" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:38:43.170335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.170284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0242520b-6e9e-485f-b3f0-a4278e0403fc" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:38:43.172053 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.172029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p" (OuterVolumeSpecName: "kube-api-access-94d4p") pod "0242520b-6e9e-485f-b3f0-a4278e0403fc" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc"). InnerVolumeSpecName "kube-api-access-94d4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:38:43.172143 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.172112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0242520b-6e9e-485f-b3f0-a4278e0403fc" (UID: "0242520b-6e9e-485f-b3f0-a4278e0403fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:38:43.270987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.270941    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0242520b-6e9e-485f-b3f0-a4278e0403fc-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:38:43.270987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.270979    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0242520b-6e9e-485f-b3f0-a4278e0403fc-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:38:43.270987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.270989    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:38:43.270987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.270999    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94d4p\" (UniqueName: \"kubernetes.io/projected/0242520b-6e9e-485f-b3f0-a4278e0403fc-kube-api-access-94d4p\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:38:43.271259 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:43.271008    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0242520b-6e9e-485f-b3f0-a4278e0403fc-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:38:44.066507 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:44.066479    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"
Sep 09 21:38:44.088110 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:44.088075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:38:44.093995 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:44.093968    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-4p2d6"]
Sep 09 21:38:45.030129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:45.030092    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" path="/var/lib/kubelet/pods/0242520b-6e9e-485f-b3f0-a4278e0403fc/volumes"
Sep 09 21:38:52.040610 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:38:52.040560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:39:02.041352 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:02.041308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:39:12.040967 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:12.040926    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 21:39:22.042141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:22.042111    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:39:24.344131 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344089    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-agent"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344414    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-agent"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344431    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344437    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344447    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344462    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="storage-initializer"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344467    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="storage-initializer"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344511    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kube-rbac-proxy"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344519    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-container"
Sep 09 21:39:24.344640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.344527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0242520b-6e9e-485f-b3f0-a4278e0403fc" containerName="kserve-agent"
Sep 09 21:39:24.347516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.347492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.350231 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.350186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 09 21:39:24.350516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.350494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:39:24.350612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.350580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 09 21:39:24.366071 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.366038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:24.407335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.407301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.407335 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.407336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.407558 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.407358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.407558 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.407422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspxz\" (UniqueName: \"kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.507965 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.507924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.507965 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.507967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.508227 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.507986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.508227 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.508021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspxz\" (UniqueName: \"kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.508403 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.508380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.508658 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.508638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.510578 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.510556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.517022 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.517000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspxz\" (UniqueName: \"kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.658032 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.657999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:24.930656 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:24.930621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:24.933185 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:39:24.933147    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48bf5eb_9426_44ec_a38c_48e33d679b61.slice/crio-5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3 WatchSource:0}: Error finding container 5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3: Status 404 returned error can't find the container with id 5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3
Sep 09 21:39:25.224537 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:25.224434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerStarted","Data":"1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"}
Sep 09 21:39:25.224537 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:25.224504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerStarted","Data":"5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3"}
Sep 09 21:39:29.241899 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.241872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/0.log"
Sep 09 21:39:29.242264 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.241915    2581 generic.go:356] "Generic (PLEG): container finished" podID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerID="1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965" exitCode=1
Sep 09 21:39:29.242264 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.241970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerDied","Data":"1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"}
Sep 09 21:39:29.267404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.267371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:39:29.267621 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.267600    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" containerID="cri-o://86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b" gracePeriod=30
Sep 09 21:39:29.268217 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:29.267727    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kube-rbac-proxy" containerID="cri-o://895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f" gracePeriod=30
Sep 09 21:39:30.247805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:30.247775    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/0.log"
Sep 09 21:39:30.248364 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:30.247852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerStarted","Data":"ebaa9cdc988635f6521c1a0fafcae57f262bdd1e5e4391d7c405d906fe90ae87"}
Sep 09 21:39:30.250004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:30.249981    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerID="895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f" exitCode=2
Sep 09 21:39:30.250120 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:30.250052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerDied","Data":"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"}
Sep 09 21:39:32.127862 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.127832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:39:32.172159 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config\") pod \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") "
Sep 09 21:39:32.172159 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm\") pod \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") "
Sep 09 21:39:32.172383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location\") pod \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") "
Sep 09 21:39:32.172383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert\") pod \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") "
Sep 09 21:39:32.172383 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls\") pod \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\" (UID: \"1f92c9cf-1064-4b64-8511-ea2f1a45da58\") "
Sep 09 21:39:32.172580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f92c9cf-1064-4b64-8511-ea2f1a45da58" (UID: "1f92c9cf-1064-4b64-8511-ea2f1a45da58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:32.172666 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.172581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f92c9cf-1064-4b64-8511-ea2f1a45da58" (UID: "1f92c9cf-1064-4b64-8511-ea2f1a45da58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:32.174440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.174413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f92c9cf-1064-4b64-8511-ea2f1a45da58" (UID: "1f92c9cf-1064-4b64-8511-ea2f1a45da58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:32.174440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.174415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm" (OuterVolumeSpecName: "kube-api-access-wj2bm") pod "1f92c9cf-1064-4b64-8511-ea2f1a45da58" (UID: "1f92c9cf-1064-4b64-8511-ea2f1a45da58"). InnerVolumeSpecName "kube-api-access-wj2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:32.182603 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.182575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f92c9cf-1064-4b64-8511-ea2f1a45da58" (UID: "1f92c9cf-1064-4b64-8511-ea2f1a45da58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:32.260035 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.260003    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerID="86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b" exitCode=0
Sep 09 21:39:32.260229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.260078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"
Sep 09 21:39:32.260229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.260085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerDied","Data":"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"}
Sep 09 21:39:32.260229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.260168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" event={"ID":"1f92c9cf-1064-4b64-8511-ea2f1a45da58","Type":"ContainerDied","Data":"b6ab7e81bbb2d91af382429715195c6c40bdc22de35b0ede2432d9d422f7e89b"}
Sep 09 21:39:32.260229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.260205    2581 scope.go:117] "RemoveContainer" containerID="895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"
Sep 09 21:39:32.269038 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.269013    2581 scope.go:117] "RemoveContainer" containerID="86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"
Sep 09 21:39:32.273611 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.273588    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:32.273611 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.273609    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kube-api-access-wj2bm\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:32.273775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.273620    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f92c9cf-1064-4b64-8511-ea2f1a45da58-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:32.273775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.273633    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f92c9cf-1064-4b64-8511-ea2f1a45da58-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:32.273775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.273648    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f92c9cf-1064-4b64-8511-ea2f1a45da58-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:32.276687 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.276661    2581 scope.go:117] "RemoveContainer" containerID="5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"
Sep 09 21:39:32.283456 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.283430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:39:32.284563 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.284540    2581 scope.go:117] "RemoveContainer" containerID="895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"
Sep 09 21:39:32.284910 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:32.284881    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f\": container with ID starting with 895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f not found: ID does not exist" containerID="895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"
Sep 09 21:39:32.285010 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.284921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f"} err="failed to get container status \"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f\": rpc error: code = NotFound desc = could not find container \"895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f\": container with ID starting with 895551b7fe38c9f1622e796e639f5899a0b7422f778dd595bb6e24649801118f not found: ID does not exist"
Sep 09 21:39:32.285010 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.284947    2581 scope.go:117] "RemoveContainer" containerID="86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"
Sep 09 21:39:32.285230 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:32.285214    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b\": container with ID starting with 86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b not found: ID does not exist" containerID="86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"
Sep 09 21:39:32.285292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.285238    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b"} err="failed to get container status \"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b\": rpc error: code = NotFound desc = could not find container \"86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b\": container with ID starting with 86ae1fa5f2346b4c3b9b7352c91fea1544cd74670b793577066bcf4c37440b5b not found: ID does not exist"
Sep 09 21:39:32.285292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.285259    2581 scope.go:117] "RemoveContainer" containerID="5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"
Sep 09 21:39:32.285514 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:32.285496    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f\": container with ID starting with 5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f not found: ID does not exist" containerID="5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"
Sep 09 21:39:32.285556 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.285520    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f"} err="failed to get container status \"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f\": rpc error: code = NotFound desc = could not find container \"5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f\": container with ID starting with 5734220484d046046690f2559fccc1287c1ab15ac79803c87a809438faf2627f not found: ID does not exist"
Sep 09 21:39:32.287953 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:32.287933    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89"]
Sep 09 21:39:33.031788 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.031759    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" path="/var/lib/kubelet/pods/1f92c9cf-1064-4b64-8511-ea2f1a45da58/volumes"
Sep 09 21:39:33.036387 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.036355    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": context deadline exceeded"
Sep 09 21:39:33.041608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.041578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmm89" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: i/o timeout"
Sep 09 21:39:33.265765 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.265660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/1.log"
Sep 09 21:39:33.266173 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.266100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/0.log"
Sep 09 21:39:33.266173 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.266132    2581 generic.go:356] "Generic (PLEG): container finished" podID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerID="ebaa9cdc988635f6521c1a0fafcae57f262bdd1e5e4391d7c405d906fe90ae87" exitCode=1
Sep 09 21:39:33.266276 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.266199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerDied","Data":"ebaa9cdc988635f6521c1a0fafcae57f262bdd1e5e4391d7c405d906fe90ae87"}
Sep 09 21:39:33.266276 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.266259    2581 scope.go:117] "RemoveContainer" containerID="1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"
Sep 09 21:39:33.266626 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.266596    2581 scope.go:117] "RemoveContainer" containerID="1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"
Sep 09 21:39:33.276688 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:33.276664    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_kserve-ci-e2e-test(e48bf5eb-9426-44ec-a38c-48e33d679b61)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61"
Sep 09 21:39:33.276809 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:33.276778    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_kserve-ci-e2e-test_e48bf5eb-9426-44ec-a38c-48e33d679b61_0 in pod sandbox 5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3 from index: no such id: '1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965'" containerID="1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"
Sep 09 21:39:33.276875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:33.276824    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_kserve-ci-e2e-test_e48bf5eb-9426-44ec-a38c-48e33d679b61_0 in pod sandbox 5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3 from index: no such id: '1b16b1e9dc0ddc15ed1e11d37d438fdcfebe6df206ff3512c1a47e860e1d1965'"
Sep 09 21:39:34.272403 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.272370    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/1.log"
Sep 09 21:39:34.413280 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.413246    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:34.543381 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543271    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:39:34.543684 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kube-rbac-proxy"
Sep 09 21:39:34.543773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543685    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kube-rbac-proxy"
Sep 09 21:39:34.543773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="storage-initializer"
Sep 09 21:39:34.543773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543747    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="storage-initializer"
Sep 09 21:39:34.543773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container"
Sep 09 21:39:34.543773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543773    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container"
Sep 09 21:39:34.543931 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543852    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kube-rbac-proxy"
Sep 09 21:39:34.543931 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.543860    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f92c9cf-1064-4b64-8511-ea2f1a45da58" containerName="kserve-container"
Sep 09 21:39:34.548244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.548222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.553115 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.553083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 09 21:39:34.553249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.553126    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:39:34.553249 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.553211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:39:34.553502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.553483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:39:34.554330 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.554315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/1.log"
Sep 09 21:39:34.556900 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.556884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:34.560000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.559980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:39:34.592375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config\") pod \"e48bf5eb-9426-44ec-a38c-48e33d679b61\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") "
Sep 09 21:39:34.592375 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location\") pod \"e48bf5eb-9426-44ec-a38c-48e33d679b61\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") "
Sep 09 21:39:34.592684 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls\") pod \"e48bf5eb-9426-44ec-a38c-48e33d679b61\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") "
Sep 09 21:39:34.592684 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hspxz\" (UniqueName: \"kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz\") pod \"e48bf5eb-9426-44ec-a38c-48e33d679b61\" (UID: \"e48bf5eb-9426-44ec-a38c-48e33d679b61\") "
Sep 09 21:39:34.592852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.592852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.592852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.592985 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e48bf5eb-9426-44ec-a38c-48e33d679b61" (UID: "e48bf5eb-9426-44ec-a38c-48e33d679b61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:34.593293 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.592895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdjd\" (UniqueName: \"kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.593293 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.593192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.593293 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.593260    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e48bf5eb-9426-44ec-a38c-48e33d679b61" (UID: "e48bf5eb-9426-44ec-a38c-48e33d679b61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:34.593480 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.593276    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e48bf5eb-9426-44ec-a38c-48e33d679b61-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:34.595067 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.595046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz" (OuterVolumeSpecName: "kube-api-access-hspxz") pod "e48bf5eb-9426-44ec-a38c-48e33d679b61" (UID: "e48bf5eb-9426-44ec-a38c-48e33d679b61"). InnerVolumeSpecName "kube-api-access-hspxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:34.595153 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.595132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e48bf5eb-9426-44ec-a38c-48e33d679b61" (UID: "e48bf5eb-9426-44ec-a38c-48e33d679b61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:34.693870 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.693827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.693870 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.693880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdjd\" (UniqueName: \"kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.693922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.693983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:34.694083    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694139    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:34.694165 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:39:34.694157    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls podName:8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d nodeName:}" failed. No retries permitted until 2026-09-09 21:39:35.194137075 +0000 UTC m=+2854.690411109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-2b8vh" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 09 21:39:34.694464 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694180    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e48bf5eb-9426-44ec-a38c-48e33d679b61-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:34.694464 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694197    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspxz\" (UniqueName: \"kubernetes.io/projected/e48bf5eb-9426-44ec-a38c-48e33d679b61-kube-api-access-hspxz\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:39:34.694464 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694681 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.694755 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.694741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:34.705340 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:34.705309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdjd\" (UniqueName: \"kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:35.197865 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.197830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:35.200220 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.200190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2b8vh\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:35.277627 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.277599    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n_e48bf5eb-9426-44ec-a38c-48e33d679b61/storage-initializer/1.log"
Sep 09 21:39:35.278048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.277711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n" event={"ID":"e48bf5eb-9426-44ec-a38c-48e33d679b61","Type":"ContainerDied","Data":"5e07c99b6709b1e09faa70ae44d218a052ff3f49a735ad1d1a21f674dc42fbd3"}
Sep 09 21:39:35.278048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.277719    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"
Sep 09 21:39:35.278048 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.277769    2581 scope.go:117] "RemoveContainer" containerID="ebaa9cdc988635f6521c1a0fafcae57f262bdd1e5e4391d7c405d906fe90ae87"
Sep 09 21:39:35.318760 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.318666    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:35.331075 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.331040    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-tlf8n"]
Sep 09 21:39:35.462961 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.462924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:35.745613 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:35.745461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:39:35.748403 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:39:35.748372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa9c5c0_5f38_4c50_b93f_9b1d0640b51d.slice/crio-d948f4c17bce35d74700b25a2583541b12f5743cc37e7b7c2d58ed8e67442d9c WatchSource:0}: Error finding container d948f4c17bce35d74700b25a2583541b12f5743cc37e7b7c2d58ed8e67442d9c: Status 404 returned error can't find the container with id d948f4c17bce35d74700b25a2583541b12f5743cc37e7b7c2d58ed8e67442d9c
Sep 09 21:39:36.283987 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:36.283953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerStarted","Data":"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"}
Sep 09 21:39:36.284376 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:36.284013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerStarted","Data":"d948f4c17bce35d74700b25a2583541b12f5743cc37e7b7c2d58ed8e67442d9c"}
Sep 09 21:39:37.029673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:37.029641    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" path="/var/lib/kubelet/pods/e48bf5eb-9426-44ec-a38c-48e33d679b61/volumes"
Sep 09 21:39:41.305286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:41.305252    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerID="69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560" exitCode=0
Sep 09 21:39:41.305710 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:41.305316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerDied","Data":"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"}
Sep 09 21:39:42.311031 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:42.310988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerStarted","Data":"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"}
Sep 09 21:39:42.311546 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:42.311063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerStarted","Data":"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"}
Sep 09 21:39:42.311546 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:42.311290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:42.334464 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:42.334407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podStartSLOduration=3.328386604 podStartE2EDuration="8.334389664s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.00600306s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:42.332307033 +0000 UTC m=+2861.828581116" watchObservedRunningTime="2026-09-09 21:39:42.334389664 +0000 UTC m=+2861.830663706"
Sep 09 21:39:43.314673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:43.314590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:43.315890 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:43.315864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:39:44.318627 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:44.318587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:39:49.324483 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:49.324452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:39:49.325147 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:49.325118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:39:59.325862 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:39:59.325774    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:40:09.325537 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:09.325492    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:40:19.325440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:19.325397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 21:40:29.325871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:29.325840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:40:35.658156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:35.658120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:40:35.658634 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:35.658377    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container" containerID="cri-o://7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3" gracePeriod=30
Sep 09 21:40:35.658780 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:35.658746    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kube-rbac-proxy" containerID="cri-o://d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10" gracePeriod=30
Sep 09 21:40:36.520210 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:36.520176    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerID="d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10" exitCode=2
Sep 09 21:40:36.520386 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:36.520253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerDied","Data":"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"}
Sep 09 21:40:38.508027 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.507999    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:40:38.529578 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.529547    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerID="7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3" exitCode=0
Sep 09 21:40:38.529748 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.529604    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"
Sep 09 21:40:38.529748 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.529624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerDied","Data":"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"}
Sep 09 21:40:38.529748 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.529692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh" event={"ID":"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d","Type":"ContainerDied","Data":"d948f4c17bce35d74700b25a2583541b12f5743cc37e7b7c2d58ed8e67442d9c"}
Sep 09 21:40:38.529873 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.529751    2581 scope.go:117] "RemoveContainer" containerID="d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"
Sep 09 21:40:38.544235 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.539384    2581 scope.go:117] "RemoveContainer" containerID="7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"
Sep 09 21:40:38.552392 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.552370    2581 scope.go:117] "RemoveContainer" containerID="69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"
Sep 09 21:40:38.560124 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.560101    2581 scope.go:117] "RemoveContainer" containerID="d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"
Sep 09 21:40:38.560435 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:40:38.560411    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10\": container with ID starting with d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10 not found: ID does not exist" containerID="d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"
Sep 09 21:40:38.560501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.560446    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10"} err="failed to get container status \"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10\": rpc error: code = NotFound desc = could not find container \"d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10\": container with ID starting with d0af76b3dd06679c72f2389a48e974e08039840f89e31f8fae230418fa64ed10 not found: ID does not exist"
Sep 09 21:40:38.560501 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.560468    2581 scope.go:117] "RemoveContainer" containerID="7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"
Sep 09 21:40:38.560781 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:40:38.560762    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3\": container with ID starting with 7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3 not found: ID does not exist" containerID="7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"
Sep 09 21:40:38.560842 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.560788    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3"} err="failed to get container status \"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3\": rpc error: code = NotFound desc = could not find container \"7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3\": container with ID starting with 7721e7c49b4e6015b350a917cbc7d0c2bdb503cfd84abe124b7b326abe85f6e3 not found: ID does not exist"
Sep 09 21:40:38.560842 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.560806    2581 scope.go:117] "RemoveContainer" containerID="69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"
Sep 09 21:40:38.561084 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:40:38.561067    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560\": container with ID starting with 69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560 not found: ID does not exist" containerID="69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"
Sep 09 21:40:38.561141 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.561090    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560"} err="failed to get container status \"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560\": rpc error: code = NotFound desc = could not find container \"69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560\": container with ID starting with 69caa0c392d9e2e78055f5be4e2dee357d9f854f7863d2c3ca0f49cf70b6f560 not found: ID does not exist"
Sep 09 21:40:38.591685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.591601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxdjd\" (UniqueName: \"kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd\") pod \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") "
Sep 09 21:40:38.591685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.591656    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert\") pod \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") "
Sep 09 21:40:38.591685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.591683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location\") pod \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") "
Sep 09 21:40:38.591926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.591720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") pod \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") "
Sep 09 21:40:38.591926 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.591760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config\") pod \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\" (UID: \"8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d\") "
Sep 09 21:40:38.592071 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.592045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:38.592210 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.592176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:38.593952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.593921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:38.593952 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.593938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd" (OuterVolumeSpecName: "kube-api-access-bxdjd") pod "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d"). InnerVolumeSpecName "kube-api-access-bxdjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:38.602597 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.602563    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" (UID: "8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:38.693236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.693184    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:40:38.693236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.693230    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdjd\" (UniqueName: \"kubernetes.io/projected/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kube-api-access-bxdjd\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:40:38.693236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.693242    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:40:38.693236 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.693250    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:40:38.693515 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.693260    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:40:38.863238 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.863095    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:40:38.865185 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:38.865160    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2b8vh"]
Sep 09 21:40:39.029682 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:39.029646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" path="/var/lib/kubelet/pods/8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d/volumes"
Sep 09 21:40:40.935145 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935109    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935362    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kube-rbac-proxy"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935373    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kube-rbac-proxy"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935386    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935408    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935433    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935476    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kube-rbac-proxy"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935482    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.935508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935488    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9c5c0-5f38-4c50-b93f-9b1d0640b51d" containerName="kserve-container"
Sep 09 21:40:40.935954 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.935565    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48bf5eb-9426-44ec-a38c-48e33d679b61" containerName="storage-initializer"
Sep 09 21:40:40.939782 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.939765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:40.943800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.943776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 09 21:40:40.943912 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.943781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:40:40.944750 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.944729    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:40:40.944854 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.944760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:40:40.944916 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.944872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:40:40.945495 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.945476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:40:40.945586 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.945532    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:40:40.955112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:40.955085    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:40:41.010954 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.010909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.010954 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.010954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.011214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.010977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.011214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.011063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.011214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.011101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112291 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112291 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112564 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112564 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:40:41.112415    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 09 21:40:41.112564 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112564 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.112564 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:40:41.112476    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls podName:de824cf4-2348-45ae-aa25-9532d5957b80 nodeName:}" failed. No retries permitted until 2026-09-09 21:40:41.612456633 +0000 UTC m=+2921.108730659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 09 21:40:41.112909 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.112888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.113166 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.113150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.113204 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.113178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.125783 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.125757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.616860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.616804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.619385 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.619358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:41.849992 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:41.849943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:42.132068 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:42.132041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:40:42.134414 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:40:42.134375    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde824cf4_2348_45ae_aa25_9532d5957b80.slice/crio-d0df34aaec6ccc1a3349a22d89f792a8dec527a69adbda4a3d117f1636a4b48b WatchSource:0}: Error finding container d0df34aaec6ccc1a3349a22d89f792a8dec527a69adbda4a3d117f1636a4b48b: Status 404 returned error can't find the container with id d0df34aaec6ccc1a3349a22d89f792a8dec527a69adbda4a3d117f1636a4b48b
Sep 09 21:40:42.553201 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:42.553116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerStarted","Data":"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"}
Sep 09 21:40:42.553348 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:42.553214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerStarted","Data":"d0df34aaec6ccc1a3349a22d89f792a8dec527a69adbda4a3d117f1636a4b48b"}
Sep 09 21:40:47.573717 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:47.573623    2581 generic.go:356] "Generic (PLEG): container finished" podID="de824cf4-2348-45ae-aa25-9532d5957b80" containerID="142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440" exitCode=0
Sep 09 21:40:47.574068 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:47.573714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerDied","Data":"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"}
Sep 09 21:40:48.580060 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.580024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerStarted","Data":"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"}
Sep 09 21:40:48.580485 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.580120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerStarted","Data":"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"}
Sep 09 21:40:48.580485 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.580395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:48.580611 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.580539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:48.581768 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.581743    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:40:48.608609 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:48.608563    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podStartSLOduration=3.5450390179999998 podStartE2EDuration="8.608547595s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.063508577s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:40:48.606912596 +0000 UTC m=+2928.103186657" watchObservedRunningTime="2026-09-09 21:40:48.608547595 +0000 UTC m=+2928.104821638"
Sep 09 21:40:49.584150 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:49.584106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:40:54.588869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:54.588838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:40:54.589529 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:54.589494    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:40:56.626269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.626235    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:40:56.630761 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.630743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.634425 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.634401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 09 21:40:56.634739 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.634724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 09 21:40:56.635520 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.635507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:40:56.644673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.644645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:40:56.730615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.730576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.730615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.730613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.730887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.730739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpptm\" (UniqueName: \"kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.730887 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.730795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832023 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.831985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.832033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.832110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpptm\" (UniqueName: \"kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.832153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.832548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.832888 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.832866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.834514 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.834495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.840830 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.840806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpptm\" (UniqueName: \"kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:56.940548 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:56.940509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:40:57.216021 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:57.215933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:40:57.219100 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:40:57.219071    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60462aa_e7fa_4ebd_9a9b_622e9f14706f.slice/crio-352d3fc8c750f903f5fa82f19037c145654c1a4283fa07d963d9ac9416083698 WatchSource:0}: Error finding container 352d3fc8c750f903f5fa82f19037c145654c1a4283fa07d963d9ac9416083698: Status 404 returned error can't find the container with id 352d3fc8c750f903f5fa82f19037c145654c1a4283fa07d963d9ac9416083698
Sep 09 21:40:57.613594 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:57.613505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerStarted","Data":"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"}
Sep 09 21:40:57.613594 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:40:57.613565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerStarted","Data":"352d3fc8c750f903f5fa82f19037c145654c1a4283fa07d963d9ac9416083698"}
Sep 09 21:41:01.630565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:01.630480    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/0.log"
Sep 09 21:41:01.630565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:01.630520    2581 generic.go:356] "Generic (PLEG): container finished" podID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerID="75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba" exitCode=1
Sep 09 21:41:01.630565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:01.630550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerDied","Data":"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"}
Sep 09 21:41:02.637298 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:02.637270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/0.log"
Sep 09 21:41:02.637720 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:02.637356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerStarted","Data":"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"}
Sep 09 21:41:04.589621 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:04.589582    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:41:06.520526 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:06.520490    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:41:06.520971 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:06.520692    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer" containerID="cri-o://99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc" gracePeriod=30
Sep 09 21:41:07.062316 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.062293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/1.log"
Sep 09 21:41:07.062746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.062730    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/0.log"
Sep 09 21:41:07.065129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.065113    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:41:07.112230 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config\") pod \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") "
Sep 09 21:41:07.112230 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112212    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location\") pod \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") "
Sep 09 21:41:07.112419 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls\") pod \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") "
Sep 09 21:41:07.112419 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpptm\" (UniqueName: \"kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm\") pod \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\" (UID: \"a60462aa-e7fa-4ebd-9a9b-622e9f14706f\") "
Sep 09 21:41:07.112491 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a60462aa-e7fa-4ebd-9a9b-622e9f14706f" (UID: "a60462aa-e7fa-4ebd-9a9b-622e9f14706f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:07.112569 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.112541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a60462aa-e7fa-4ebd-9a9b-622e9f14706f" (UID: "a60462aa-e7fa-4ebd-9a9b-622e9f14706f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:07.114352 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.114326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a60462aa-e7fa-4ebd-9a9b-622e9f14706f" (UID: "a60462aa-e7fa-4ebd-9a9b-622e9f14706f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:07.114451 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.114349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm" (OuterVolumeSpecName: "kube-api-access-vpptm") pod "a60462aa-e7fa-4ebd-9a9b-622e9f14706f" (UID: "a60462aa-e7fa-4ebd-9a9b-622e9f14706f"). InnerVolumeSpecName "kube-api-access-vpptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:07.213175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.213123    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:07.213175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.213168    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:07.213175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.213178    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:07.213175 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.213187    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpptm\" (UniqueName: \"kubernetes.io/projected/a60462aa-e7fa-4ebd-9a9b-622e9f14706f-kube-api-access-vpptm\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:07.636674 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.636634    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.636934    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.636945    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.636955    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.636961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.637024    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.637078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.637030    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerName="storage-initializer"
Sep 09 21:41:07.640255 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.640235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.643286 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.643260    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 09 21:41:07.643417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.643274    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:41:07.654665 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.654639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:41:07.658315 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/1.log"
Sep 09 21:41:07.658735 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658721    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42_a60462aa-e7fa-4ebd-9a9b-622e9f14706f/storage-initializer/0.log"
Sep 09 21:41:07.658830 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658754    2581 generic.go:356] "Generic (PLEG): container finished" podID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" containerID="99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc" exitCode=1
Sep 09 21:41:07.658882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerDied","Data":"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"}
Sep 09 21:41:07.658882 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42" event={"ID":"a60462aa-e7fa-4ebd-9a9b-622e9f14706f","Type":"ContainerDied","Data":"352d3fc8c750f903f5fa82f19037c145654c1a4283fa07d963d9ac9416083698"}
Sep 09 21:41:07.658983 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658901    2581 scope.go:117] "RemoveContainer" containerID="99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"
Sep 09 21:41:07.658983 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.658830    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"
Sep 09 21:41:07.668806 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.668785    2581 scope.go:117] "RemoveContainer" containerID="75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"
Sep 09 21:41:07.682750 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.682723    2581 scope.go:117] "RemoveContainer" containerID="99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"
Sep 09 21:41:07.683095 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:07.683070    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc\": container with ID starting with 99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc not found: ID does not exist" containerID="99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"
Sep 09 21:41:07.683186 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.683109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc"} err="failed to get container status \"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc\": rpc error: code = NotFound desc = could not find container \"99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc\": container with ID starting with 99ec57ed933aab961d9a0e63a5e4dacd2ac4f2807c0560ea3f0c69e5ef49b1cc not found: ID does not exist"
Sep 09 21:41:07.683186 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.683135    2581 scope.go:117] "RemoveContainer" containerID="75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"
Sep 09 21:41:07.683401 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:07.683382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba\": container with ID starting with 75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba not found: ID does not exist" containerID="75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"
Sep 09 21:41:07.683438 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.683409    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba"} err="failed to get container status \"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba\": rpc error: code = NotFound desc = could not find container \"75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba\": container with ID starting with 75aab6fff1d252755dd9c22763b7203c8c0400dfd5189df4584f30aad23887ba not found: ID does not exist"
Sep 09 21:41:07.706717 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.706670    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:41:07.711140 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.711112    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-4nc42"]
Sep 09 21:41:07.716746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.716718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.716891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.716770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf8t\" (UniqueName: \"kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.716891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.716841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.716989 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.716938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.717041 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.716989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818040 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.817998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf8t\" (UniqueName: \"kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818040 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818522 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818921 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.818921 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.818914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.820692 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.820666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.826483 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.826460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf8t\" (UniqueName: \"kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t\") pod \"isvc-pmml-runtime-predictor-885fc8777-p6gnr\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:07.950860 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:07.950816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:08.224519 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:08.224429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:41:08.228070 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:41:08.228040    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01aa24f_1cbd_4cdb_a9a3_84a36b5e6761.slice/crio-41de2eb689cfbe1fccd3bc76858a6e917cf060e119eb18e530f44f896fe4e637 WatchSource:0}: Error finding container 41de2eb689cfbe1fccd3bc76858a6e917cf060e119eb18e530f44f896fe4e637: Status 404 returned error can't find the container with id 41de2eb689cfbe1fccd3bc76858a6e917cf060e119eb18e530f44f896fe4e637
Sep 09 21:41:08.665078 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:08.665022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerStarted","Data":"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"}
Sep 09 21:41:08.665444 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:08.665114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerStarted","Data":"41de2eb689cfbe1fccd3bc76858a6e917cf060e119eb18e530f44f896fe4e637"}
Sep 09 21:41:09.030766 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:09.030676    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60462aa-e7fa-4ebd-9a9b-622e9f14706f" path="/var/lib/kubelet/pods/a60462aa-e7fa-4ebd-9a9b-622e9f14706f/volumes"
Sep 09 21:41:12.682423 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:12.682389    2581 generic.go:356] "Generic (PLEG): container finished" podID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerID="97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13" exitCode=0
Sep 09 21:41:12.682844 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:12.682441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerDied","Data":"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"}
Sep 09 21:41:14.589588 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:14.589432    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:41:20.723567 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:20.723523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerStarted","Data":"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"}
Sep 09 21:41:20.724018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:20.723611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerStarted","Data":"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"}
Sep 09 21:41:20.724018 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:20.723867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:20.747247 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:20.747182    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podStartSLOduration=2.790012781 podStartE2EDuration="13.74716261s" totalImagesPullingTime="7.30105329s" totalInitContainerRuntime="3.656096539s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:20.74622841 +0000 UTC m=+2960.242502490" watchObservedRunningTime="2026-09-09 21:41:20.74716261 +0000 UTC m=+2960.243436653"
Sep 09 21:41:21.728277 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:21.728227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:21.729769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:21.729726    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:41:22.732804 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:22.732758    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:41:24.590272 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:24.590230    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:41:27.737630 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:27.737600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:41:27.738179 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:27.738147    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:41:34.589902 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:34.589868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:41:37.738940 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:37.738897    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:41:42.253943 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:42.253909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:41:42.254416 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:42.254230    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kube-rbac-proxy" containerID="cri-o://ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c" gracePeriod=30
Sep 09 21:41:42.254416 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:42.254216    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" containerID="cri-o://c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94" gracePeriod=30
Sep 09 21:41:42.808398 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:42.808366    2581 generic.go:356] "Generic (PLEG): container finished" podID="de824cf4-2348-45ae-aa25-9532d5957b80" containerID="ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c" exitCode=2
Sep 09 21:41:42.808579 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:42.808441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerDied","Data":"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"}
Sep 09 21:41:44.584670 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:44.584627    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 09 21:41:44.589977 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:44.589945    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 21:41:45.095918 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.095893    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:41:45.227191 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6\") pod \"de824cf4-2348-45ae-aa25-9532d5957b80\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") "
Sep 09 21:41:45.227191 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location\") pod \"de824cf4-2348-45ae-aa25-9532d5957b80\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") "
Sep 09 21:41:45.227191 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert\") pod \"de824cf4-2348-45ae-aa25-9532d5957b80\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") "
Sep 09 21:41:45.227484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227213    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config\") pod \"de824cf4-2348-45ae-aa25-9532d5957b80\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") "
Sep 09 21:41:45.227484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") pod \"de824cf4-2348-45ae-aa25-9532d5957b80\" (UID: \"de824cf4-2348-45ae-aa25-9532d5957b80\") "
Sep 09 21:41:45.227619 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227591    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de824cf4-2348-45ae-aa25-9532d5957b80" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:45.227674 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.227610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de824cf4-2348-45ae-aa25-9532d5957b80" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:45.229322 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.229294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de824cf4-2348-45ae-aa25-9532d5957b80" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:45.229442 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.229364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6" (OuterVolumeSpecName: "kube-api-access-qpdx6") pod "de824cf4-2348-45ae-aa25-9532d5957b80" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80"). InnerVolumeSpecName "kube-api-access-qpdx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:45.238639 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.238608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de824cf4-2348-45ae-aa25-9532d5957b80" (UID: "de824cf4-2348-45ae-aa25-9532d5957b80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:45.328339 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.328302    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de824cf4-2348-45ae-aa25-9532d5957b80-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:45.328339 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.328335    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:45.328496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.328346    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de824cf4-2348-45ae-aa25-9532d5957b80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:45.328496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.328355    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de824cf4-2348-45ae-aa25-9532d5957b80-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:45.328496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.328364    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdx6\" (UniqueName: \"kubernetes.io/projected/de824cf4-2348-45ae-aa25-9532d5957b80-kube-api-access-qpdx6\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:41:45.821490 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.821451    2581 generic.go:356] "Generic (PLEG): container finished" podID="de824cf4-2348-45ae-aa25-9532d5957b80" containerID="c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94" exitCode=0
Sep 09 21:41:45.821964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.821505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerDied","Data":"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"}
Sep 09 21:41:45.821964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.821561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6" event={"ID":"de824cf4-2348-45ae-aa25-9532d5957b80","Type":"ContainerDied","Data":"d0df34aaec6ccc1a3349a22d89f792a8dec527a69adbda4a3d117f1636a4b48b"}
Sep 09 21:41:45.821964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.821508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"
Sep 09 21:41:45.821964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.821602    2581 scope.go:117] "RemoveContainer" containerID="ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"
Sep 09 21:41:45.831019 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.830995    2581 scope.go:117] "RemoveContainer" containerID="c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"
Sep 09 21:41:45.838608 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.838588    2581 scope.go:117] "RemoveContainer" containerID="142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"
Sep 09 21:41:45.846384 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.846361    2581 scope.go:117] "RemoveContainer" containerID="ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"
Sep 09 21:41:45.846717 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:45.846676    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c\": container with ID starting with ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c not found: ID does not exist" containerID="ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"
Sep 09 21:41:45.846802 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.846726    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c"} err="failed to get container status \"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c\": rpc error: code = NotFound desc = could not find container \"ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c\": container with ID starting with ad1310faa5d71547ee5c0d9f7c16c852c556e2b8130a2235faa95f64d174ba0c not found: ID does not exist"
Sep 09 21:41:45.846802 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.846751    2581 scope.go:117] "RemoveContainer" containerID="c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"
Sep 09 21:41:45.846802 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.846766    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:41:45.847013 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:45.846998    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94\": container with ID starting with c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94 not found: ID does not exist" containerID="c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"
Sep 09 21:41:45.847050 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.847016    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94"} err="failed to get container status \"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94\": rpc error: code = NotFound desc = could not find container \"c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94\": container with ID starting with c2552f72e5c3ce95ee26949821799432df392af8d1c1fe1750a0d48649d4ff94 not found: ID does not exist"
Sep 09 21:41:45.847050 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.847030    2581 scope.go:117] "RemoveContainer" containerID="142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"
Sep 09 21:41:45.847291 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:45.847277    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440\": container with ID starting with 142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440 not found: ID does not exist" containerID="142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"
Sep 09 21:41:45.847336 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.847294    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440"} err="failed to get container status \"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440\": rpc error: code = NotFound desc = could not find container \"142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440\": container with ID starting with 142c4de3fb1af6668dcb55e42a2dbe8d77d2c8f55bb0a443043acd13c1056440 not found: ID does not exist"
Sep 09 21:41:45.851883 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:45.851860    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fg9f6"]
Sep 09 21:41:47.029549 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.029509    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" path="/var/lib/kubelet/pods/de824cf4-2348-45ae-aa25-9532d5957b80/volumes"
Sep 09 21:41:47.510685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.510639    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:41:47.511004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.510975    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kube-rbac-proxy"
Sep 09 21:41:47.511004 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.510993    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kube-rbac-proxy"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511008    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511029    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="storage-initializer"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="storage-initializer"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511109    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kserve-container"
Sep 09 21:41:47.511156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.511121    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de824cf4-2348-45ae-aa25-9532d5957b80" containerName="kube-rbac-proxy"
Sep 09 21:41:47.515842 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.515818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.520126 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.520099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 09 21:41:47.520547 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.520530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 09 21:41:47.529261 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.529232    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:41:47.648538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.648493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.648538 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.648540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.648775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.648573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdsx\" (UniqueName: \"kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.648775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.648597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.648775 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.648716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.738757 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.738722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:41:47.750131 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750221 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750221 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdsx\" (UniqueName: \"kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750221 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750346 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:47.750328    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 09 21:41:47.750447 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:41:47.750397    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls podName:fb7fb76f-9d81-4894-af9a-0a64b12206ef nodeName:}" failed. No retries permitted until 2026-09-09 21:41:48.250376905 +0000 UTC m=+2987.746650945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls") pod "isvc-pmml-predictor-8674795f59-9zp25" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 09 21:41:47.750719 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.750871 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.751000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.750984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:47.762214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:47.762152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdsx\" (UniqueName: \"kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:48.253999 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.253955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:48.256424 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.256400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9zp25\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:48.430316 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.430272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:48.714034 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.713999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:41:48.718924 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:41:48.718893    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7fb76f_9d81_4894_af9a_0a64b12206ef.slice/crio-6dda5c5b4fb449f5df631b8233a1ac14eb95f98759b9948aa97da9cfeec87185 WatchSource:0}: Error finding container 6dda5c5b4fb449f5df631b8233a1ac14eb95f98759b9948aa97da9cfeec87185: Status 404 returned error can't find the container with id 6dda5c5b4fb449f5df631b8233a1ac14eb95f98759b9948aa97da9cfeec87185
Sep 09 21:41:48.840407 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.840373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerStarted","Data":"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"}
Sep 09 21:41:48.840613 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:48.840438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerStarted","Data":"6dda5c5b4fb449f5df631b8233a1ac14eb95f98759b9948aa97da9cfeec87185"}
Sep 09 21:41:52.857659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:52.857566    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerID="7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8" exitCode=0
Sep 09 21:41:52.857659 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:52.857641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerDied","Data":"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"}
Sep 09 21:41:53.864163 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:53.864126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerStarted","Data":"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"}
Sep 09 21:41:53.864536 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:53.864198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerStarted","Data":"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"}
Sep 09 21:41:53.864536 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:53.864387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:53.884483 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:53.884424    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podStartSLOduration=3.174741463 podStartE2EDuration="6.884407874s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.709666411s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:53.882185765 +0000 UTC m=+2993.378459831" watchObservedRunningTime="2026-09-09 21:41:53.884407874 +0000 UTC m=+2993.380681941"
Sep 09 21:41:54.867869 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:54.867834    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:41:54.869206 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:54.869179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:41:55.871762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:55.871720    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:41:57.738600 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:41:57.738560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:00.877263 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:00.877232    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:42:00.877740 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:00.877716    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:01.572244 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:01.572212    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:42:01.573042 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:01.573021    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:42:07.738125 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:07.738080    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:10.878579 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:10.878541    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:17.738815 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:17.738773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:20.878449 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:20.878408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:27.738250 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:27.738207    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:29.025120 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:29.025077    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:30.877735 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:30.877671    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:39.025505 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:39.025448    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:40.877928 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:40.877884    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:49.029510 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:49.029479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:42:50.878196 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:50.878156    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:42:58.858449 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:58.858371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:42:58.858996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:58.858603    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" containerID="cri-o://6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101" gracePeriod=30
Sep 09 21:42:58.858996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:58.858635    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kube-rbac-proxy" containerID="cri-o://9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba" gracePeriod=30
Sep 09 21:42:59.025179 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:59.025139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 21:42:59.130314 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:59.130234    2581 generic.go:356] "Generic (PLEG): container finished" podID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerID="9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba" exitCode=2
Sep 09 21:42:59.130314 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:42:59.130274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerDied","Data":"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"}
Sep 09 21:43:00.878490 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:00.878455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:43:02.807072 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.807040    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:43:02.955015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.954963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert\") pod \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") "
Sep 09 21:43:02.955015 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls\") pod \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") "
Sep 09 21:43:02.955270 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location\") pod \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") "
Sep 09 21:43:02.955270 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config\") pod \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") "
Sep 09 21:43:02.955270 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf8t\" (UniqueName: \"kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t\") pod \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\" (UID: \"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761\") "
Sep 09 21:43:02.955448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" (UID: "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:43:02.955516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" (UID: "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:02.955565 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.955533    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" (UID: "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:02.957252 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.957224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t" (OuterVolumeSpecName: "kube-api-access-6zf8t") pod "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" (UID: "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761"). InnerVolumeSpecName "kube-api-access-6zf8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:02.957341 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:02.957306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" (UID: "e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:03.056649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.056608    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:03.056649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.056641    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:03.056649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.056651    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:03.056649 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.056660    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:03.056937 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.056671    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf8t\" (UniqueName: \"kubernetes.io/projected/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761-kube-api-access-6zf8t\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:03.148717 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.148676    2581 generic.go:356] "Generic (PLEG): container finished" podID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerID="6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101" exitCode=0
Sep 09 21:43:03.148915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.148735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerDied","Data":"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"}
Sep 09 21:43:03.148915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.148762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"
Sep 09 21:43:03.148915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.148802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" event={"ID":"e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761","Type":"ContainerDied","Data":"41de2eb689cfbe1fccd3bc76858a6e917cf060e119eb18e530f44f896fe4e637"}
Sep 09 21:43:03.148915 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.148833    2581 scope.go:117] "RemoveContainer" containerID="9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"
Sep 09 21:43:03.157362 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.157344    2581 scope.go:117] "RemoveContainer" containerID="6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"
Sep 09 21:43:03.165165 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.165144    2581 scope.go:117] "RemoveContainer" containerID="97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"
Sep 09 21:43:03.168129 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.168105    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:43:03.172513 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.172489    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr"]
Sep 09 21:43:03.173630 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.173614    2581 scope.go:117] "RemoveContainer" containerID="9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"
Sep 09 21:43:03.173977 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:03.173946    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba\": container with ID starting with 9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba not found: ID does not exist" containerID="9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"
Sep 09 21:43:03.174040 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.173985    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba"} err="failed to get container status \"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba\": rpc error: code = NotFound desc = could not find container \"9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba\": container with ID starting with 9eec349368d26facf446479d0bef7cf023eb42ba0eb81250700b286a87c349ba not found: ID does not exist"
Sep 09 21:43:03.174040 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.174004    2581 scope.go:117] "RemoveContainer" containerID="6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"
Sep 09 21:43:03.174267 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:03.174252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101\": container with ID starting with 6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101 not found: ID does not exist" containerID="6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"
Sep 09 21:43:03.174355 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.174271    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101"} err="failed to get container status \"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101\": rpc error: code = NotFound desc = could not find container \"6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101\": container with ID starting with 6dcde63229a6168beda447e769201a4eacb95be2fef8c390f2996c132eb34101 not found: ID does not exist"
Sep 09 21:43:03.174355 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.174290    2581 scope.go:117] "RemoveContainer" containerID="97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"
Sep 09 21:43:03.174518 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:03.174500    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13\": container with ID starting with 97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13 not found: ID does not exist" containerID="97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"
Sep 09 21:43:03.174570 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.174527    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13"} err="failed to get container status \"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13\": rpc error: code = NotFound desc = could not find container \"97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13\": container with ID starting with 97686acfec5af8101585137f8b49706a6f49b6d5f8054bb845e34a0587847a13 not found: ID does not exist"
Sep 09 21:43:03.734288 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:03.734243    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p6gnr" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:43:04.042573 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042496    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042797    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="storage-initializer"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042809    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="storage-initializer"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042818    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042823    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042831    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kube-rbac-proxy"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kube-rbac-proxy"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042926    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kube-rbac-proxy"
Sep 09 21:43:04.043001 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.042943    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" containerName="kserve-container"
Sep 09 21:43:04.047733 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.047690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.050744 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.050722    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:43:04.051991 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.051971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 09 21:43:04.058117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.058095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:43:04.166973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.166933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.166973 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.166983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48psk\" (UniqueName: \"kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.167214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.167041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.167214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.167062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.167214 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.167089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.267808 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.267766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.267808 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.267810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268157 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.267832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268157 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.267862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268157 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.267890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48psk\" (UniqueName: \"kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268337 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.268242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268624 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.268602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.268737 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.268644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.270241 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.270223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.276933 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.276905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48psk\" (UniqueName: \"kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.358613 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.358513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:04.637213 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:04.637187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:43:04.639799 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:43:04.639769    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1e9c6e_781f_4858_84e5_570fd2ee30e8.slice/crio-f8f28fda128a7983b6c69d77ed7e7ec046fcb1220ba1bfcf01c392616a1b6447 WatchSource:0}: Error finding container f8f28fda128a7983b6c69d77ed7e7ec046fcb1220ba1bfcf01c392616a1b6447: Status 404 returned error can't find the container with id f8f28fda128a7983b6c69d77ed7e7ec046fcb1220ba1bfcf01c392616a1b6447
Sep 09 21:43:05.029258 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:05.029228    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761" path="/var/lib/kubelet/pods/e01aa24f-1cbd-4cdb-a9a3-84a36b5e6761/volumes"
Sep 09 21:43:05.160642 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:05.160606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerStarted","Data":"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"}
Sep 09 21:43:05.161049 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:05.160664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerStarted","Data":"f8f28fda128a7983b6c69d77ed7e7ec046fcb1220ba1bfcf01c392616a1b6447"}
Sep 09 21:43:09.179044 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:09.179012    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerID="75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef" exitCode=0
Sep 09 21:43:09.179439 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:09.179050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerDied","Data":"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"}
Sep 09 21:43:10.184865 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:10.184829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerStarted","Data":"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"}
Sep 09 21:43:10.185247 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:10.184923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerStarted","Data":"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"}
Sep 09 21:43:10.185247 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:10.185120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:10.208331 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:10.208274    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podStartSLOduration=2.556376636 podStartE2EDuration="6.208257965s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.651881329s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:10.206156854 +0000 UTC m=+3069.702430921" watchObservedRunningTime="2026-09-09 21:43:10.208257965 +0000 UTC m=+3069.704532006"
Sep 09 21:43:10.878278 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:10.878240    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 21:43:11.189189 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:11.189159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:11.190612 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:11.190582    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:12.192685 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:12.192644    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:17.029762 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:17.029731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:43:17.197640 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:17.197607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:43:17.198194 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:17.198160    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:18.370319 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:18.370286    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:43:18.370778 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:18.370492    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container" containerID="cri-o://4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3" gracePeriod=30
Sep 09 21:43:18.370778 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:18.370544    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kube-rbac-proxy" containerID="cri-o://a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01" gracePeriod=30
Sep 09 21:43:19.221859 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:19.221821    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerID="a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01" exitCode=2
Sep 09 21:43:19.222050 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:19.221887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerDied","Data":"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"}
Sep 09 21:43:20.872289 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:20.872251    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 09 21:43:22.214161 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.214131    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:43:22.236665 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.236631    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerID="4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3" exitCode=0
Sep 09 21:43:22.236874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.236728    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"
Sep 09 21:43:22.236874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.236692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerDied","Data":"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"}
Sep 09 21:43:22.236874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.236813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25" event={"ID":"fb7fb76f-9d81-4894-af9a-0a64b12206ef","Type":"ContainerDied","Data":"6dda5c5b4fb449f5df631b8233a1ac14eb95f98759b9948aa97da9cfeec87185"}
Sep 09 21:43:22.236874 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.236862    2581 scope.go:117] "RemoveContainer" containerID="a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"
Sep 09 21:43:22.245516 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.245494    2581 scope.go:117] "RemoveContainer" containerID="4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"
Sep 09 21:43:22.254547 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.254527    2581 scope.go:117] "RemoveContainer" containerID="7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"
Sep 09 21:43:22.263200 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.263182    2581 scope.go:117] "RemoveContainer" containerID="a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"
Sep 09 21:43:22.263500 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:22.263480    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01\": container with ID starting with a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01 not found: ID does not exist" containerID="a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"
Sep 09 21:43:22.263572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.263509    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01"} err="failed to get container status \"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01\": rpc error: code = NotFound desc = could not find container \"a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01\": container with ID starting with a29527db06f567718a4883b6c61cf206c9d4a2064597737bf43e9fe934125f01 not found: ID does not exist"
Sep 09 21:43:22.263572 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.263527    2581 scope.go:117] "RemoveContainer" containerID="4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"
Sep 09 21:43:22.263821 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:22.263802    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3\": container with ID starting with 4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3 not found: ID does not exist" containerID="4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"
Sep 09 21:43:22.263868 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.263827    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3"} err="failed to get container status \"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3\": rpc error: code = NotFound desc = could not find container \"4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3\": container with ID starting with 4e29cc14ba7bddc85be174f976b6e39294d274dac6cb7237d09f47e0d59ab0e3 not found: ID does not exist"
Sep 09 21:43:22.263868 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.263843    2581 scope.go:117] "RemoveContainer" containerID="7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"
Sep 09 21:43:22.264137 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:43:22.264116    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8\": container with ID starting with 7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8 not found: ID does not exist" containerID="7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"
Sep 09 21:43:22.264192 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.264148    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8"} err="failed to get container status \"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8\": rpc error: code = NotFound desc = could not find container \"7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8\": container with ID starting with 7ecc54973af64d64fd57d90e8d763dbc698dfe426caecc395f02ee76a58277a8 not found: ID does not exist"
Sep 09 21:43:22.301655 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.301572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") pod \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") "
Sep 09 21:43:22.301655 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.301642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcdsx\" (UniqueName: \"kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx\") pod \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") "
Sep 09 21:43:22.301877 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.301662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config\") pod \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") "
Sep 09 21:43:22.301877 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.301682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert\") pod \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") "
Sep 09 21:43:22.301877 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.301727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location\") pod \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\" (UID: \"fb7fb76f-9d81-4894-af9a-0a64b12206ef\") "
Sep 09 21:43:22.302082 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.302056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb7fb76f-9d81-4894-af9a-0a64b12206ef" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:22.302082 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.302077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb7fb76f-9d81-4894-af9a-0a64b12206ef" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:43:22.302233 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.302052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb7fb76f-9d81-4894-af9a-0a64b12206ef" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:22.303793 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.303773    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx" (OuterVolumeSpecName: "kube-api-access-lcdsx") pod "fb7fb76f-9d81-4894-af9a-0a64b12206ef" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef"). InnerVolumeSpecName "kube-api-access-lcdsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:22.303875 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.303821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb7fb76f-9d81-4894-af9a-0a64b12206ef" (UID: "fb7fb76f-9d81-4894-af9a-0a64b12206ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:22.402631 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.402587    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb7fb76f-9d81-4894-af9a-0a64b12206ef-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:22.402631 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.402627    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdsx\" (UniqueName: \"kubernetes.io/projected/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-api-access-lcdsx\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:22.402631 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.402641    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:22.402904 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.402655    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb7fb76f-9d81-4894-af9a-0a64b12206ef-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:22.402904 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.402669    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb7fb76f-9d81-4894-af9a-0a64b12206ef-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:43:22.564187 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.564098    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:43:22.569121 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:22.569088    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9zp25"]
Sep 09 21:43:23.029496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:23.029458    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" path="/var/lib/kubelet/pods/fb7fb76f-9d81-4894-af9a-0a64b12206ef/volumes"
Sep 09 21:43:27.199062 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:27.199025    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:37.198744 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:37.198692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:47.199049 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:47.199007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:43:57.199079 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:43:57.199039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:44:07.198496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:07.198459    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:44:17.198942 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:17.198901    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:44:23.025829 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:23.025780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 09 21:44:33.029147 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:33.029115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:44:35.167110 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:35.167074    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:44:35.167524 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:35.167304    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container" containerID="cri-o://6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde" gracePeriod=30
Sep 09 21:44:35.167524 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:35.167369    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kube-rbac-proxy" containerID="cri-o://c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf" gracePeriod=30
Sep 09 21:44:35.511690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:35.511603    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerID="c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf" exitCode=2
Sep 09 21:44:35.511690 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:35.511655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerDied","Data":"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"}
Sep 09 21:44:37.193858 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:37.193813    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 09 21:44:38.914469 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:38.914442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:44:39.107803 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.107720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config\") pod \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") "
Sep 09 21:44:39.107803 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.107784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location\") pod \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") "
Sep 09 21:44:39.108012 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.107808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert\") pod \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") "
Sep 09 21:44:39.108012 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.107832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48psk\" (UniqueName: \"kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk\") pod \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") "
Sep 09 21:44:39.108012 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.107879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls\") pod \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\" (UID: \"ca1e9c6e-781f-4858-84e5-570fd2ee30e8\") "
Sep 09 21:44:39.108149 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.108121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca1e9c6e-781f-4858-84e5-570fd2ee30e8" (UID: "ca1e9c6e-781f-4858-84e5-570fd2ee30e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:39.108193 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.108145    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca1e9c6e-781f-4858-84e5-570fd2ee30e8" (UID: "ca1e9c6e-781f-4858-84e5-570fd2ee30e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:39.108238 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.108214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca1e9c6e-781f-4858-84e5-570fd2ee30e8" (UID: "ca1e9c6e-781f-4858-84e5-570fd2ee30e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:39.109964 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.109937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca1e9c6e-781f-4858-84e5-570fd2ee30e8" (UID: "ca1e9c6e-781f-4858-84e5-570fd2ee30e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:39.110090 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.109972    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk" (OuterVolumeSpecName: "kube-api-access-48psk") pod "ca1e9c6e-781f-4858-84e5-570fd2ee30e8" (UID: "ca1e9c6e-781f-4858-84e5-570fd2ee30e8"). InnerVolumeSpecName "kube-api-access-48psk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:39.209448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.209411    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:44:39.209448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.209443    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:44:39.209448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.209454    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:44:39.209686 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.209463    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:44:39.209686 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.209474    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48psk\" (UniqueName: \"kubernetes.io/projected/ca1e9c6e-781f-4858-84e5-570fd2ee30e8-kube-api-access-48psk\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:44:39.528599 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.528565    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerID="6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde" exitCode=0
Sep 09 21:44:39.528801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.528634    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"
Sep 09 21:44:39.528801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.528654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerDied","Data":"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"}
Sep 09 21:44:39.528801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.528737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr" event={"ID":"ca1e9c6e-781f-4858-84e5-570fd2ee30e8","Type":"ContainerDied","Data":"f8f28fda128a7983b6c69d77ed7e7ec046fcb1220ba1bfcf01c392616a1b6447"}
Sep 09 21:44:39.528801 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.528769    2581 scope.go:117] "RemoveContainer" containerID="c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"
Sep 09 21:44:39.537168 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.537149    2581 scope.go:117] "RemoveContainer" containerID="6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"
Sep 09 21:44:39.544621 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.544605    2581 scope.go:117] "RemoveContainer" containerID="75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"
Sep 09 21:44:39.552161 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.552133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:44:39.552517 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.552494    2581 scope.go:117] "RemoveContainer" containerID="c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"
Sep 09 21:44:39.552845 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:44:39.552817    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf\": container with ID starting with c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf not found: ID does not exist" containerID="c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"
Sep 09 21:44:39.552923 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.552858    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf"} err="failed to get container status \"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf\": rpc error: code = NotFound desc = could not find container \"c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf\": container with ID starting with c1f6275a7d2d144fa229883c1bde42ad314ccfb92b76e81f6aacde0f4a1313bf not found: ID does not exist"
Sep 09 21:44:39.552923 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.552876    2581 scope.go:117] "RemoveContainer" containerID="6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"
Sep 09 21:44:39.553160 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:44:39.553135    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde\": container with ID starting with 6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde not found: ID does not exist" containerID="6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"
Sep 09 21:44:39.553209 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.553167    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde"} err="failed to get container status \"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde\": rpc error: code = NotFound desc = could not find container \"6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde\": container with ID starting with 6a638320dda5e81712e798d81ed650298de947ba3d5b62f40f4ceb270acbffde not found: ID does not exist"
Sep 09 21:44:39.553209 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.553183    2581 scope.go:117] "RemoveContainer" containerID="75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"
Sep 09 21:44:39.553473 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:44:39.553454    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef\": container with ID starting with 75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef not found: ID does not exist" containerID="75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"
Sep 09 21:44:39.553528 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.553477    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef"} err="failed to get container status \"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef\": rpc error: code = NotFound desc = could not find container \"75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef\": container with ID starting with 75434bee22af390ebb4d0d258ea35c0bc98e44c1ab3281675fcdf79657a702ef not found: ID does not exist"
Sep 09 21:44:39.555497 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:39.555475    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-24tmr"]
Sep 09 21:44:40.371945 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.371910    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372170    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372181    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372206    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372212    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372221    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="storage-initializer"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372226    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="storage-initializer"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372237    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="storage-initializer"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372243    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="storage-initializer"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372253    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372258    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372299    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372305    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kserve-container"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372310    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7fb76f-9d81-4894-af9a-0a64b12206ef" containerName="kube-rbac-proxy"
Sep 09 21:44:40.372326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.372316    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" containerName="kserve-container"
Sep 09 21:44:40.376673 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.376649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.379800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.379776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:44:40.379800 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.379775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:44:40.380065 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.379806    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:44:40.380065 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.379825    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-db4d1e-kube-rbac-proxy-sar-config\""
Sep 09 21:44:40.380065 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.379963    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:44:40.381025 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.381006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-db4d1e-predictor-serving-cert\""
Sep 09 21:44:40.381283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.381245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:44:40.388307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.388286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:44:40.418763 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.418713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.418958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.418777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.418958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.418848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.418958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.418878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxcz\" (UniqueName: \"kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.418958 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.418916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.519626 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.519585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.519626 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.519633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.519891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.519669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.519891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.519687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxcz\" (UniqueName: \"kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.519891 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.519728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.520116 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.520091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.520508 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.520487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.520605 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.520509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.522282 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.522252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.530269 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.530244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxcz\" (UniqueName: \"kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz\") pod \"isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") " pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.686888 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.686854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:40.957494 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:40.957358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:44:40.960115 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:44:40.960087    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e86d569_a724_4029_b2ff_9150fac27e0a.slice/crio-bd21007775de69dd7b2d1ef5239d61a95e5ebbdd8d5e99527c9aa2aa951d958f WatchSource:0}: Error finding container bd21007775de69dd7b2d1ef5239d61a95e5ebbdd8d5e99527c9aa2aa951d958f: Status 404 returned error can't find the container with id bd21007775de69dd7b2d1ef5239d61a95e5ebbdd8d5e99527c9aa2aa951d958f
Sep 09 21:44:41.030395 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:41.030363    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1e9c6e-781f-4858-84e5-570fd2ee30e8" path="/var/lib/kubelet/pods/ca1e9c6e-781f-4858-84e5-570fd2ee30e8/volumes"
Sep 09 21:44:41.539417 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:41.539377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerStarted","Data":"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"}
Sep 09 21:44:41.539805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:41.539436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerStarted","Data":"bd21007775de69dd7b2d1ef5239d61a95e5ebbdd8d5e99527c9aa2aa951d958f"}
Sep 09 21:44:45.555059 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:45.555017    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerID="ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba" exitCode=0
Sep 09 21:44:45.555446 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:45.555093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerDied","Data":"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"}
Sep 09 21:44:46.561156 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.561121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerStarted","Data":"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"}
Sep 09 21:44:46.561525 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.561182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerStarted","Data":"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"}
Sep 09 21:44:46.561525 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.561443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:46.561647 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.561576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:46.562799 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.562775    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:44:46.584623 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:46.584573    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podStartSLOduration=2.591036228 podStartE2EDuration="6.584554445s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993518217s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:44:46.583230874 +0000 UTC m=+3166.079504932" watchObservedRunningTime="2026-09-09 21:44:46.584554445 +0000 UTC m=+3166.080828552"
Sep 09 21:44:47.565463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:47.565410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:44:52.569880 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:52.569846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:44:52.570426 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:44:52.570398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:02.571245 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:02.571201    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:12.570635 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:12.570595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:22.570823 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:22.570787    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:32.571298 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:32.571258    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:42.570360 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:42.570307    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:45:52.570562 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:45:52.570524    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:46:00.025931 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:00.025839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 21:46:10.026545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:10.026512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:46:25.605117 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.605077    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:25.608404 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.608381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.611042 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.611014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-db4d1e\""
Sep 09 21:46:25.611176 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.611074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-db4d1e-predictor-serving-cert\""
Sep 09 21:46:25.611176 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.611088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-db4d1e-dockercfg-gmp54\""
Sep 09 21:46:25.611176 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.611089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-db4d1e-kube-rbac-proxy-sar-config\""
Sep 09 21:46:25.621032 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.621007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:25.754462 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.754420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.754462 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.754465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.754746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.754500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.754746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.754564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.754746 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.754612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz287\" (UniqueName: \"kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855112 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855369 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz287\" (UniqueName: \"kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855580 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.855949 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.856000 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.855939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.857748 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.857723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.864641 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.864618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz287\" (UniqueName: \"kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287\") pod \"isvc-secondary-db4d1e-predictor-5859965988-nvsck\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") " pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:25.918577 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:25.918538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:26.195639 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:26.195613    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:26.197733 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:46:26.197688    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233c0fcd_4f18_46d1_b006_17b3c3bb6145.slice/crio-6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7 WatchSource:0}: Error finding container 6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7: Status 404 returned error can't find the container with id 6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7
Sep 09 21:46:26.936484 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:26.936451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerStarted","Data":"8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"}
Sep 09 21:46:26.936893 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:26.936506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerStarted","Data":"6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7"}
Sep 09 21:46:32.959632 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:32.959606    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/0.log"
Sep 09 21:46:32.960038 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:32.959643    2581 generic.go:356] "Generic (PLEG): container finished" podID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerID="8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c" exitCode=1
Sep 09 21:46:32.960038 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:32.959719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerDied","Data":"8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"}
Sep 09 21:46:33.966120 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:33.966090    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/0.log"
Sep 09 21:46:33.966602 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:33.966154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerStarted","Data":"a50110e6efdd408c1f25342a12b3707a937b16a1eec6103b0b9fb7ab9e7431cd"}
Sep 09 21:46:38.986094 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.986066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/1.log"
Sep 09 21:46:38.986500 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.986469    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/0.log"
Sep 09 21:46:38.986545 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.986504    2581 generic.go:356] "Generic (PLEG): container finished" podID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerID="a50110e6efdd408c1f25342a12b3707a937b16a1eec6103b0b9fb7ab9e7431cd" exitCode=1
Sep 09 21:46:38.986625 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.986579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerDied","Data":"a50110e6efdd408c1f25342a12b3707a937b16a1eec6103b0b9fb7ab9e7431cd"}
Sep 09 21:46:38.986731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.986671    2581 scope.go:117] "RemoveContainer" containerID="8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"
Sep 09 21:46:38.987023 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.987004    2581 scope.go:117] "RemoveContainer" containerID="8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"
Sep 09 21:46:38.997138 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:46:38.997104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-db4d1e-predictor-5859965988-nvsck_kserve-ci-e2e-test(233c0fcd-4f18-46d1-b006-17b3c3bb6145)\"" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145"
Sep 09 21:46:38.997364 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:46:38.997329    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-db4d1e-predictor-5859965988-nvsck_kserve-ci-e2e-test_233c0fcd-4f18-46d1-b006-17b3c3bb6145_0 in pod sandbox 6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7 from index: no such id: '8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c'" containerID="8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"
Sep 09 21:46:38.997448 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:38.997381    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-db4d1e-predictor-5859965988-nvsck_kserve-ci-e2e-test_233c0fcd-4f18-46d1-b006-17b3c3bb6145_0 in pod sandbox 6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7 from index: no such id: '8935bc83caa5ddd7f68298f60b1413c3c27e992ee9552a69eaf2fc71cc9e931c'"
Sep 09 21:46:39.991855 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:39.991827    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/1.log"
Sep 09 21:46:43.588996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.588960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:43.625513 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.625480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:46:43.625814 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.625787    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container" containerID="cri-o://a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52" gracePeriod=30
Sep 09 21:46:43.625928 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.625815    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kube-rbac-proxy" containerID="cri-o://164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936" gracePeriod=30
Sep 09 21:46:43.744250 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.744232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/1.log"
Sep 09 21:46:43.746346 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.746331    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:43.904604 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.904572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config\") pod \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") "
Sep 09 21:46:43.904805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.904638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert\") pod \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") "
Sep 09 21:46:43.904805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.904672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location\") pod \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") "
Sep 09 21:46:43.904805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.904726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls\") pod \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") "
Sep 09 21:46:43.904805 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.904758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz287\" (UniqueName: \"kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287\") pod \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\" (UID: \"233c0fcd-4f18-46d1-b006-17b3c3bb6145\") "
Sep 09 21:46:43.905047 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.905010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "233c0fcd-4f18-46d1-b006-17b3c3bb6145" (UID: "233c0fcd-4f18-46d1-b006-17b3c3bb6145"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:43.905113 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.905083    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "233c0fcd-4f18-46d1-b006-17b3c3bb6145" (UID: "233c0fcd-4f18-46d1-b006-17b3c3bb6145"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:43.905180 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.905109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "233c0fcd-4f18-46d1-b006-17b3c3bb6145" (UID: "233c0fcd-4f18-46d1-b006-17b3c3bb6145"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:43.906881 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.906853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "233c0fcd-4f18-46d1-b006-17b3c3bb6145" (UID: "233c0fcd-4f18-46d1-b006-17b3c3bb6145"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:43.907009 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:43.906960    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287" (OuterVolumeSpecName: "kube-api-access-pz287") pod "233c0fcd-4f18-46d1-b006-17b3c3bb6145" (UID: "233c0fcd-4f18-46d1-b006-17b3c3bb6145"). InnerVolumeSpecName "kube-api-access-pz287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:44.005496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.005461    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:44.005496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.005487    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:44.005496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.005497    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/233c0fcd-4f18-46d1-b006-17b3c3bb6145-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:44.005496 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.005507    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz287\" (UniqueName: \"kubernetes.io/projected/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-api-access-pz287\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:44.005924 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.005518    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/233c0fcd-4f18-46d1-b006-17b3c3bb6145-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:44.008155 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.008134    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-db4d1e-predictor-5859965988-nvsck_233c0fcd-4f18-46d1-b006-17b3c3bb6145/storage-initializer/1.log"
Sep 09 21:46:44.008292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.008231    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"
Sep 09 21:46:44.008292 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.008263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck" event={"ID":"233c0fcd-4f18-46d1-b006-17b3c3bb6145","Type":"ContainerDied","Data":"6cfffff8979d3b9b340203dd94f2c895a8d6cc4d6db54251fea2fd3d65072ab7"}
Sep 09 21:46:44.008387 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.008339    2581 scope.go:117] "RemoveContainer" containerID="a50110e6efdd408c1f25342a12b3707a937b16a1eec6103b0b9fb7ab9e7431cd"
Sep 09 21:46:44.010652 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.010626    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerID="164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936" exitCode=2
Sep 09 21:46:44.010815 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.010665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerDied","Data":"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"}
Sep 09 21:46:44.050587 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.050548    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:44.056647 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:44.056616    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-db4d1e-predictor-5859965988-nvsck"]
Sep 09 21:46:45.030130 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:45.030084    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" path="/var/lib/kubelet/pods/233c0fcd-4f18-46d1-b006-17b3c3bb6145/volumes"
Sep 09 21:46:47.565947 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:47.565904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 09 21:46:48.370670 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.370645    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:46:48.543093 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.542997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls\") pod \"6e86d569-a724-4029-b2ff-9150fac27e0a\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") "
Sep 09 21:46:48.543093 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert\") pod \"6e86d569-a724-4029-b2ff-9150fac27e0a\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") "
Sep 09 21:46:48.543307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmxcz\" (UniqueName: \"kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz\") pod \"6e86d569-a724-4029-b2ff-9150fac27e0a\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") "
Sep 09 21:46:48.543307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location\") pod \"6e86d569-a724-4029-b2ff-9150fac27e0a\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") "
Sep 09 21:46:48.543307 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config\") pod \"6e86d569-a724-4029-b2ff-9150fac27e0a\" (UID: \"6e86d569-a724-4029-b2ff-9150fac27e0a\") "
Sep 09 21:46:48.543518 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e86d569-a724-4029-b2ff-9150fac27e0a" (UID: "6e86d569-a724-4029-b2ff-9150fac27e0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:48.543591 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e86d569-a724-4029-b2ff-9150fac27e0a" (UID: "6e86d569-a724-4029-b2ff-9150fac27e0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:48.543591 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.543559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e86d569-a724-4029-b2ff-9150fac27e0a" (UID: "6e86d569-a724-4029-b2ff-9150fac27e0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:48.545252 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.545229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e86d569-a724-4029-b2ff-9150fac27e0a" (UID: "6e86d569-a724-4029-b2ff-9150fac27e0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:48.545388 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.545260    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz" (OuterVolumeSpecName: "kube-api-access-cmxcz") pod "6e86d569-a724-4029-b2ff-9150fac27e0a" (UID: "6e86d569-a724-4029-b2ff-9150fac27e0a"). InnerVolumeSpecName "kube-api-access-cmxcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:48.644226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.644184    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-cabundle-cert\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:48.644226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.644217    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxcz\" (UniqueName: \"kubernetes.io/projected/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-api-access-cmxcz\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:48.644226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.644227    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e86d569-a724-4029-b2ff-9150fac27e0a-kserve-provision-location\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:48.644226 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.644235    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e86d569-a724-4029-b2ff-9150fac27e0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:48.644731 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:48.644245    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e86d569-a724-4029-b2ff-9150fac27e0a-proxy-tls\") on node \"ip-10-0-137-145.ec2.internal\" DevicePath \"\""
Sep 09 21:46:49.032324 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.032290    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerID="a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52" exitCode=0
Sep 09 21:46:49.032502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.032345    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"
Sep 09 21:46:49.032502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.032374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerDied","Data":"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"}
Sep 09 21:46:49.032502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.032439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k" event={"ID":"6e86d569-a724-4029-b2ff-9150fac27e0a","Type":"ContainerDied","Data":"bd21007775de69dd7b2d1ef5239d61a95e5ebbdd8d5e99527c9aa2aa951d958f"}
Sep 09 21:46:49.032502 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.032479    2581 scope.go:117] "RemoveContainer" containerID="164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"
Sep 09 21:46:49.041449 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.041431    2581 scope.go:117] "RemoveContainer" containerID="a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"
Sep 09 21:46:49.050957 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.050913    2581 scope.go:117] "RemoveContainer" containerID="ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"
Sep 09 21:46:49.058521 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.058488    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:46:49.059752 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.059731    2581 scope.go:117] "RemoveContainer" containerID="164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"
Sep 09 21:46:49.060116 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:46:49.060096    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936\": container with ID starting with 164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936 not found: ID does not exist" containerID="164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"
Sep 09 21:46:49.060227 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.060128    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936"} err="failed to get container status \"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936\": rpc error: code = NotFound desc = could not find container \"164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936\": container with ID starting with 164e974e150194d662f7a6abf467f77d46508185ad20756bfd2a3d731af9c936 not found: ID does not exist"
Sep 09 21:46:49.060227 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.060150    2581 scope.go:117] "RemoveContainer" containerID="a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"
Sep 09 21:46:49.060550 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:46:49.060454    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52\": container with ID starting with a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52 not found: ID does not exist" containerID="a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"
Sep 09 21:46:49.060550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.060486    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52"} err="failed to get container status \"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52\": rpc error: code = NotFound desc = could not find container \"a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52\": container with ID starting with a7ef05fe19cf40ee617e8c7b508a7ffacb6b3c6edb53a1d100d7693de0410f52 not found: ID does not exist"
Sep 09 21:46:49.060550 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.060508    2581 scope.go:117] "RemoveContainer" containerID="ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"
Sep 09 21:46:49.061266 ip-10-0-137-145 kubenswrapper[2581]: E0909 21:46:49.061237    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba\": container with ID starting with ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba not found: ID does not exist" containerID="ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"
Sep 09 21:46:49.061379 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.061272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba"} err="failed to get container status \"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba\": rpc error: code = NotFound desc = could not find container \"ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba\": container with ID starting with ca1f05237edd5c846245c506419d3246c70908fdf4bcbfd3bb0a9639acda2fba not found: ID does not exist"
Sep 09 21:46:49.062297 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:49.062281    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-db4d1e-predictor-7bfc75d6dc-bxz9k"]
Sep 09 21:46:51.030088 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:46:51.030054    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" path="/var/lib/kubelet/pods/6e86d569-a724-4029-b2ff-9150fac27e0a/volumes"
Sep 09 21:47:01.595652 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:01.595627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:47:01.596177 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:01.595684    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:47:26.416106 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416027    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qw289/must-gather-snsqf"]
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416341    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416356    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416364    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416376    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kube-rbac-proxy"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416395    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kube-rbac-proxy"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.416463 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416462    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kserve-container"
Sep 09 21:47:26.416852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416469    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.416852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416478    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e86d569-a724-4029-b2ff-9150fac27e0a" containerName="kube-rbac-proxy"
Sep 09 21:47:26.416852 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.416485    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="233c0fcd-4f18-46d1-b006-17b3c3bb6145" containerName="storage-initializer"
Sep 09 21:47:26.419304 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.419288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.422020 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.421997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qw289\"/\"default-dockercfg-x8qhr\""
Sep 09 21:47:26.422159 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.422024    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw289\"/\"kube-root-ca.crt\""
Sep 09 21:47:26.422159 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.421999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw289\"/\"openshift-service-ca.crt\""
Sep 09 21:47:26.427779 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.427747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/must-gather-snsqf"]
Sep 09 21:47:26.516888 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.516844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxcv\" (UniqueName: \"kubernetes.io/projected/fa43402f-8f72-4eaa-a429-850987858f8e-kube-api-access-czxcv\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.516888 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.516886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa43402f-8f72-4eaa-a429-850987858f8e-must-gather-output\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.617781 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.617741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxcv\" (UniqueName: \"kubernetes.io/projected/fa43402f-8f72-4eaa-a429-850987858f8e-kube-api-access-czxcv\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.617977 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.617788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa43402f-8f72-4eaa-a429-850987858f8e-must-gather-output\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.618192 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.618169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa43402f-8f72-4eaa-a429-850987858f8e-must-gather-output\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.632254 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.632223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxcv\" (UniqueName: \"kubernetes.io/projected/fa43402f-8f72-4eaa-a429-850987858f8e-kube-api-access-czxcv\") pod \"must-gather-snsqf\" (UID: \"fa43402f-8f72-4eaa-a429-850987858f8e\") " pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:26.728951 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:26.728860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/must-gather-snsqf"
Sep 09 21:47:27.003513 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:27.003432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/must-gather-snsqf"]
Sep 09 21:47:27.006716 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:47:27.006672    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa43402f_8f72_4eaa_a429_850987858f8e.slice/crio-030be6b799f378fcb3d5974d900f6203f26075b167fc3d163cb2bf45838c3615 WatchSource:0}: Error finding container 030be6b799f378fcb3d5974d900f6203f26075b167fc3d163cb2bf45838c3615: Status 404 returned error can't find the container with id 030be6b799f378fcb3d5974d900f6203f26075b167fc3d163cb2bf45838c3615
Sep 09 21:47:27.008576 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:27.008560    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:47:27.175275 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:27.175236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/must-gather-snsqf" event={"ID":"fa43402f-8f72-4eaa-a429-850987858f8e","Type":"ContainerStarted","Data":"030be6b799f378fcb3d5974d900f6203f26075b167fc3d163cb2bf45838c3615"}
Sep 09 21:47:28.181597 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:28.181548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/must-gather-snsqf" event={"ID":"fa43402f-8f72-4eaa-a429-850987858f8e","Type":"ContainerStarted","Data":"1d7fdbc782d9ddff349a736066ffde9bcf198e2f46792619a2c168297a23533f"}
Sep 09 21:47:28.182127 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:28.181610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/must-gather-snsqf" event={"ID":"fa43402f-8f72-4eaa-a429-850987858f8e","Type":"ContainerStarted","Data":"19f463e040d5f89c1b215f544cdfdb7cc1c17b3ff6ad05c81a83064ff3f4fd6b"}
Sep 09 21:47:28.200185 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:28.200132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qw289/must-gather-snsqf" podStartSLOduration=1.395634973 podStartE2EDuration="2.200113989s" totalImagesPullingTime="804.479016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:28.199356546 +0000 UTC m=+3327.695630589" watchObservedRunningTime="2026-09-09 21:47:28.200113989 +0000 UTC m=+3327.696388036"
Sep 09 21:47:29.388573 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:29.388538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qqzz4_e35bdaed-546d-4756-ae8b-d4fa6ed1aab1/global-pull-secret-syncer/0.log"
Sep 09 21:47:29.545761 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:29.545731    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bkmnv_1a8162d3-7b99-4095-8d27-0aa11f28eb5c/konnectivity-agent/0.log"
Sep 09 21:47:29.659481 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:29.659438    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-145.ec2.internal_1001eb5eb924c8ce04785f3e1fa27845/haproxy/0.log"
Sep 09 21:47:34.307475 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:34.307439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sv6df_a649a47c-7865-48b1-8cf8-0d7a06e6185e/node-exporter/0.log"
Sep 09 21:47:34.335993 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:34.335960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sv6df_a649a47c-7865-48b1-8cf8-0d7a06e6185e/kube-rbac-proxy/0.log"
Sep 09 21:47:34.366055 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:34.366028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sv6df_a649a47c-7865-48b1-8cf8-0d7a06e6185e/init-textfile/0.log"
Sep 09 21:47:36.896821 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:36.896782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/1.log"
Sep 09 21:47:36.903355 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:36.903286    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rbgh4_150c182f-77ba-4a14-9989-1fb90702c008/console-operator/2.log"
Sep 09 21:47:37.332814 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:37.332732    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-866f57cbb9-qvtl6_19f9a012-0e2b-4080-afe7-9b29e41e4d8f/console/0.log"
Sep 09 21:47:37.872489 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:37.872447    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"]
Sep 09 21:47:37.878125 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:37.878095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:37.887848 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:37.887819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"]
Sep 09 21:47:38.018828 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.018766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-lib-modules\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.019317 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.018845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-proc\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.019317 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.018918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-sys\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.019317 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.018949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-podres\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.019317 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.018987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97h6x\" (UniqueName: \"kubernetes.io/projected/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-kube-api-access-97h6x\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120005 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.119960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-podres\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120310 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-podres\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120440 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97h6x\" (UniqueName: \"kubernetes.io/projected/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-kube-api-access-97h6x\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120615 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-lib-modules\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120773 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-proc\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.120913 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-proc\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.121167 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-sys\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.121334 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-sys\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.121466 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.120811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-lib-modules\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.130052 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.129976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97h6x\" (UniqueName: \"kubernetes.io/projected/8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1-kube-api-access-97h6x\") pod \"perf-node-gather-daemonset-jdgbk\" (UID: \"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.191976 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.191920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:38.482188 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.482158    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"]
Sep 09 21:47:38.484852 ip-10-0-137-145 kubenswrapper[2581]: W0909 21:47:38.484819    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8e9bf1cf_daba_44ab_ae6c_c5100dd8d8d1.slice/crio-76392f9851884cff8f5a5f5fe52438492365486d0e6885e724a4c54ea5600734 WatchSource:0}: Error finding container 76392f9851884cff8f5a5f5fe52438492365486d0e6885e724a4c54ea5600734: Status 404 returned error can't find the container with id 76392f9851884cff8f5a5f5fe52438492365486d0e6885e724a4c54ea5600734
Sep 09 21:47:38.891996 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.891920    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rd5h_6b479bfe-ed66-450b-9f46-980dc2d8de9f/dns/0.log"
Sep 09 21:47:38.915575 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:38.915547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rd5h_6b479bfe-ed66-450b-9f46-980dc2d8de9f/kube-rbac-proxy/0.log"
Sep 09 21:47:39.044554 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.044395    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhfl6_7bb328a0-4ea0-4e63-9965-a10f79458da8/dns-node-resolver/0.log"
Sep 09 21:47:39.240074 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.240039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk" event={"ID":"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1","Type":"ContainerStarted","Data":"b8e35ac2dc3ba2a23d8f691f4b345d8ec743953213ae2ccfda995c11cb0de457"}
Sep 09 21:47:39.240293 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.240097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk" event={"ID":"8e9bf1cf-daba-44ab-ae6c-c5100dd8d8d1","Type":"ContainerStarted","Data":"76392f9851884cff8f5a5f5fe52438492365486d0e6885e724a4c54ea5600734"}
Sep 09 21:47:39.240293 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.240207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:39.269332 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.269264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk" podStartSLOduration=2.26924137 podStartE2EDuration="2.26924137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:39.269078734 +0000 UTC m=+3338.765352790" watchObservedRunningTime="2026-09-09 21:47:39.26924137 +0000 UTC m=+3338.765515428"
Sep 09 21:47:39.736521 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:39.736483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ppvcw_75d33de8-9ddb-4f97-9691-1ee5220ba6cb/node-ca/0.log"
Sep 09 21:47:40.495087 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:40.495055    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-87766bcc8-xnv4m_44b96dff-bfb5-4403-8890-2225918ddc65/router/0.log"
Sep 09 21:47:40.875691 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:40.875624    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n98lw_50f7ee11-afe2-4227-a6e0-a77d8e392718/serve-healthcheck-canary/0.log"
Sep 09 21:47:41.234315 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:41.234284    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-xnphq_4425e76b-d0e7-4308-b2da-66eb66adcc39/insights-operator/0.log"
Sep 09 21:47:41.234795 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:41.234776    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-xnphq_4425e76b-d0e7-4308-b2da-66eb66adcc39/insights-operator/1.log"
Sep 09 21:47:41.399283 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:41.399247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlxqk_c4fce036-efab-41b6-9236-535aff9d4915/kube-rbac-proxy/0.log"
Sep 09 21:47:41.423115 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:41.423085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlxqk_c4fce036-efab-41b6-9236-535aff9d4915/exporter/0.log"
Sep 09 21:47:41.445769 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:41.445742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wlxqk_c4fce036-efab-41b6-9236-535aff9d4915/extractor/0.log"
Sep 09 21:47:43.844349 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:43.844315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-644949856f-6h6v2_3f527382-9f9e-4d38-a115-609cff575578/manager/0.log"
Sep 09 21:47:45.254276 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:45.254239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-jdgbk"
Sep 09 21:47:51.056340 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.056305    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:47:51.096554 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.096525    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/egress-router-binary-copy/0.log"
Sep 09 21:47:51.135332 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.135303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/cni-plugins/0.log"
Sep 09 21:47:51.171437 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.171403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/bond-cni-plugin/0.log"
Sep 09 21:47:51.194423 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.194393    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/routeoverride-cni/0.log"
Sep 09 21:47:51.220229 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.220202    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/whereabouts-cni-bincopy/0.log"
Sep 09 21:47:51.249462 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.249430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vpgql_287066b7-5bb3-4553-9818-ffd2b8840bf9/whereabouts-cni/0.log"
Sep 09 21:47:51.294689 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.294657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m687m_1b25be79-7378-407a-88e9-068d191bd16d/kube-multus/0.log"
Sep 09 21:47:51.408665 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.408634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bzt4v_6ab21338-9de8-4bef-a83e-5e42f15a438b/network-metrics-daemon/0.log"
Sep 09 21:47:51.445290 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:51.445260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bzt4v_6ab21338-9de8-4bef-a83e-5e42f15a438b/kube-rbac-proxy/0.log"
Sep 09 21:47:53.022237 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.022198    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/ovn-controller/0.log"
Sep 09 21:47:53.041499 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.041471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/ovn-acl-logging/0.log"
Sep 09 21:47:53.082080 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.082049    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/kube-rbac-proxy-node/0.log"
Sep 09 21:47:53.106457 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.106431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:47:53.133090 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.133067    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/northd/0.log"
Sep 09 21:47:53.154326 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.154300    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/nbdb/0.log"
Sep 09 21:47:53.181100 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.181073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/sbdb/0.log"
Sep 09 21:47:53.233912 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:53.233881    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zp8rd_3b650cf8-1258-47a9-a786-3df974f4ae9c/ovnkube-controller/0.log"
Sep 09 21:47:54.350104 ip-10-0-137-145 kubenswrapper[2581]: I0909 21:47:54.350079    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qn2gw_7ec06c5e-3aef-4ea9-95a2-b4efe9f42365/network-check-target-container/0.log"